Product Requirements README
Table of Contents
Problem Statement
Gathering data from arbitrary log sources should not be a time-consuming task. Blueteams, network admins, and anyone wanting to record and store the events of a service should have an efficient, secure method of doing so.
Goals and Objectives
A stream of log events, provided by the client endpoint, will be packaged and transported to a central cloud server for storage and retrieval.
Whether on a windows or *Nix-based endpoint, a user will be able to create an account, receive proper credentials, and establish a secure connection between the source of their logs and the central data-storage server.
Compatibility with ElasticSearch should be available to the user that will complement the service the central storage server provides.
User Type and Scenarios
User |
Scenario |
Priority |
|---|---|---|
Security Analyst |
As a security analyst, I want to be able to specify relevant log sources for my investigations. |
High |
IT Adminstrator |
As an IT admin, I want an intuitive interface for configuring, initializing, and retrieving log data. |
High |
Network Administrator |
As a Network admin, I want the retrieval of network-related logs to be supported for network analysis. |
High |
Functional Requirements
Data Collection:
The system should be able to collect data from the specified source.
Specify supported data formats for efficient data parsing.
TLS Connection Establishment:
Implement TLS over TCP socket creation and management.
Create, Update, and Revoke certificates with an in-house CA manager
Data Processing and Serialization:
The collected data should be processed and serialized for transmission over the TCP connection.
Error Handling and Reliability:
Implement error detection and correction mechanisms.
TCP handles retransmission of lost or corrupted data.
Implement system monitoring and alerting to reduce downtime during system degradation
Destination Data Reception:
Develop a system of database I/O.
Create a procedure to integrate with ElasticSearch.
User Authentication:
Whitelist specific user IP addresses.
Key-value store mapping users to certificates.
Mechanism for revoking certificates if needed.
Create keys for the user to match certificate values.
Non-functional Requirements
Latency
Response times should be less than 300ms
Support a 50% increase in log traffic
Recovery
Failure recovery without data loss
Compatibility and Compliance
WCAG compliant
Data protection compliant
Device and system agnostic (windows/*nix)
Documentation
User and developer documentation
Availability and Maintenance
24/7 availability, except during scheduled maintenance periods
Installability (server/client components)
Easy to perform, fast, and well explained through documentation
Visual Representations
User Experience and Usability Plan
A first-time user will navigate to the product page (frontend repo) read about the product software, and make a decision about whether this service fits their needs. If their needs can be met with service provided, they will create an account and be redirected to their account dashboard.
From their account dashboard, they will be able to:
download their certificate and private key
download the sofware along with the license
copy their one-time-password (OTP) information (important)
view existing log sources (not the data) being collected
The user will install the software (Tauri app) onto their machine.
When that process is complete, they will run the binary (doubleclick/cli) and be presented with a login screen where they will enter their OTP (or user/pass).
When they have successfully authenticated, full visibility of log data will be available. The user will be able to interact with each log source, download their data from the central server, and add/remove log sources.
We will make every attempt to ensure compatibility with users with disabilities as the project progresses.
Technical Architecture and Choices
The service running on the client machine will be written in Rust. The interface for client machines will be written using Tauri, another Rust-based project.
The reason for Rust being its ability to write and compile to multiple operating systems and handle threading and async operations. Along with its platform versatility, Rust’s cargo command line tool has options for testing and documentation generation.
Tauri makes GUI creation simple. Given its cross-platform compatibility, due to it being written in Rust, just makes sense.
The service running on the server will be a mix of languages, such as Terraform, shell, Python, and Rust if needed. A relational database will be used, with options left open to the user for integration with ElasticSearch if they choose.
The website provides the user signup, software distribution, and documentation will use Svelte.
Timeline
An MVP of this service should be up and running at month 3. This will give 4 months for user testing/additional development.
Dependencies and Bottlenecks
Still todo but: team size, creating the in-house CA, using Rust
Testing and Quality Assurance
Between now and the 7th month, code and methods will be tested on a regular basis as outlined in the SDP document [PROVIDE LINK TO SDP README].
Test Conditions to be met:
API Repo:
All API endpoints:
Protect API endpoints through access verification
Public Key Infrastructure API endpoint must handle the following:
Issue certificates
Verify the identity of the requesting user requesting a digital signature
Revoke certificates of specific users
Renew certificates of specific users
Key Pair storage and retrieval
Certificate policy (cert lifetime, validation, key length reqs, etc.)
Follows industry standards (x509, bitlength, cipher type, etc.)
PKI audit logging
PKI hierarchy (masterkey -> prodkey1 -> [prodkey2, …, prodkeyN])
Failure Recovery (offline masterkey)
User management api endpoints must handle the following:
Create, Read, Update, and Delete users
Account management (signin/out/delete,cert retrieval/renewal, etc.)
Grant or revoke user certification.
Whitelist specific IP addresses.
Key-value to certificate matching for user authentication.
Dashboard api endpoints must handle the following:
Data visualization
Software distribution api endpoints must handle the following:
Provide links to download software for existing platforms
Frontend Repo:
Provide the interface to interact with the API
Device agnostic
Contains links to User and Developer documentation
Server Repo:
Create a TLS connection with Client
Create a database specific to the user
Each user database has a table for each input log source
Delete database on user deletion
Automate the initialization of Server Instance
Client Repo:
Create a TLS connection with Server
Command line and Graphical User Interface to configure:
the destination address
certificate (supplied from API repo)
path(s) to log source(s)
Documentation Repo:
Provides setup instructions for:
account management
certificate management
client-server connection establishment
Establish a [Changelog](SUPPLY LINK TO CHANGELOG)
Continuous Documentation Deployment here: https://securitylogminer-doc-repo.readthedocs.io/en/latest/
User Documentation and Support
User documentation will be written to satisfy the requirements of the documentation requirement section
Support for this project will come as new maintainers are added to the organization. Since this is an open-source project, forks will be possible under the allowed [license](PROVIDE LINK TO SOFTWARE LICENSE).
Documentation is setup and hosted on Read The Docs. This type of integration allows for continuous updates to the documentation, whenever a development change occurs. Whenever updates are pushed to our documentation repository on GitHub, Read the Docs automatically rebuilds the documentation, ensuring that users always have access to the most current information.
Linked documentation: https://securitylogminer-doc-repo.readthedocs.io/en/latest/
Versioning
Major: breaking changes
Minor: new features
Patch: bug fixes
Stable versions will be available on the Minor versions due to feature availability.
Feasibility
The initial team tasked with this project is capable but we recognize that this will be a considerable amount of work. A minimal viable product is not past realistic expectations.
Innovative Features
There are no features that exist within this project that do not exist elsewhere. What makes this product stand out is its availability on platforms. The service it provides could pave the way for innovative services due the product’s core data collection service.
Stakeholder Alignment
Stakeholder approves of overall project vision.
Change Management
Updates to software will be accompanied by a [Changelog](SUPPLY LINK TO CHANGELOG).
Evaluation
The success of this product will be determined by satisfying the defined testing conditions.