MLOPS documentation

Architecture-Overview:

• 1. Introduction and Goals
  • 1.1 Task Description
  • 1.2 Quality-Target
  • 1.3 Stakeholders
• 2. Constraints
  • 2.1 Technical
  • 2.2 Organization
• 3. Context and Scope
  • 3.1 Adult Income Context
  • 3.2 Technical Context
• 4. Solution Strategy
  • 4.1 Guiding Principles
  • 4.2 Architectural Approach
  • 4.3 Technology Stack
  • 4.4 Risk Management
• 5. Building Block View
  • 5.1 Overview of Building Blocks
  • 5.2 Detailed Breakdown of Components
• 6. Runtime View
• 7. Deployment View
  • 7.1 Infrastructure Overview
  • 7.2 Deployment Diagram
  • 7.3 Network and Hosting Details
  • 7.4 Deployment Process
  • 7.5 Scaling and Future Improvements
• 8. Crosscutting Concepts
  • 8.1 Security and Authentication
  • 8.2 Observability and Monitoring
  • 8.3 Logging and Tracing
  • 8.4 Configuration Management
  • 8.5 Scalability and Resilience
  • 8.6 Continuous Integration and Deployment (CI/CD)
  • 8.7 Data Versioning and Management
• 9. Architecture Decisions
  • 9.1 FastAPI for Backend Services
  • 9.2 Deployment Strategy: Docker
  • 9.3 Database Choice: PostgreSQL
  • 9.4 Storage: MinIO as S3-Compatible Object Store
  • 9.5 Monitoring & Logging: Prometheus + Grafana + Loki
  • 9.5 CI/CD Strategy: GitHub Actions for Automation
  • 9.6 API Gateway & Reverse Proxy: Nginx
• 10. Quality Requirements
• 11. Risks and Technical Debt
  • 11.1 Risks
  • 11.2 Technical Debt
• 12. Glossary

Services:

• Data-Processor
  • app package
    • Subpackages
      • app.core package
        • Subpackages
          • app.core.celery package
            • Submodules
            • app.core.celery.app module
            • app.core.celery.celeryconfig module
            • Module contents
        • Submodules
        • app.core.dvc_client module
          • DVCClient
            • DVCClient.__init__()
            • DVCClient.read_data_from()
            • DVCClient.save_data_to()
        • Module contents
      • app.tasks package
        • Subpackages
          • app.tasks.data_processing package
            • Submodules
            • app.tasks.data_processing.tasks module
            • Module contents
          • app.tasks.data_transformation package
            • Submodules
            • app.tasks.data_transformation.tasks module
            • Module contents
          • app.tasks.modeling package
            • Submodules
            • app.tasks.modeling.tasks module
            • Module contents
          • app.tasks.workflows package
            • Submodules
            • app.tasks.workflows.tasks module
            • Module contents
        • Module contents
    • Submodules
    • app.constants module
      • EnvConfig
        • EnvConfig.API_BEARER_TOKEN
        • EnvConfig.CELERY_BACKEND_CONNECTION
        • EnvConfig.CELERY_BROKER_CONNECTION
        • EnvConfig.CELERY_DEFAULT_QUEUE
        • EnvConfig.PORT
        • EnvConfig.S3_ACCESS_KEY_ID
        • EnvConfig.S3_BUCKET_NAME
        • EnvConfig.S3_ENDPOINT_URL
        • EnvConfig.S3_SECRET_ACCESS_KEY
    • app.utils module
      • get_metadata()
      • retrieve_metadata()
    • Module contents
• Streamlit-UI
  • app package
    • Submodules
    • app.backend module
    • app.constants module
      • EnvConfig
        • EnvConfig.API_BEARER_TOKEN
        • EnvConfig.CELERY_BACKEND_CONNECTION
        • EnvConfig.CELERY_BROKER_CONNECTION
        • EnvConfig.CELERY_DEFAULT_QUEUE
        • EnvConfig.PORT
        • EnvConfig.S3_ACCESS_KEY_ID
        • EnvConfig.S3_BUCKET_NAME
        • EnvConfig.S3_ENDPOINT_URL
        • EnvConfig.S3_SECRET_ACCESS_KEY
    • app.reloader module
    • app.schemas module
      • AsyncTaskResponse
        • AsyncTaskResponse.id
        • AsyncTaskResponse.result
        • AsyncTaskResponse.status
      • FeedbackInputRequest
        • FeedbackInputRequest.income
        • FeedbackInputRequest.task_id
      • FileMetadataResponse
        • FileMetadataResponse.columns
      • UserInputRequest
        • UserInputRequest.age
        • UserInputRequest.capital_gain
        • UserInputRequest.capital_loss
        • UserInputRequest.education
        • UserInputRequest.educational_num
        • UserInputRequest.fnlwgt
        • UserInputRequest.gender
        • UserInputRequest.hours_per_week
        • UserInputRequest.marital_status
        • UserInputRequest.native_country
        • UserInputRequest.occupation
        • UserInputRequest.race
        • UserInputRequest.relationship
        • UserInputRequest.workclass
    • app.ui module
    • Module contents
• Pipeline-API
  • app package
    • Subpackages
      • app.core package
        • Submodules
        • app.core.celery_client module
          • CeleryClient
            • CeleryClient.__init__()
            • CeleryClient.get_app()
            • CeleryClient.get_result()
            • CeleryClient.get_status()
            • CeleryClient.get_task()
        • app.core.data_factory module
          • DataFactory
            • DataFactory.from_bytes()
            • DataFactory.from_dict()
            • DataFactory.merge_dfs()
        • app.core.dvc_client module
          • DVCClient
            • DVCClient.__init__()
            • DVCClient.read_data_from()
            • DVCClient.save_data_to()
        • Module contents
      • app.routers package
        • Submodules
        • app.routers.api module
        • app.routers.health module
        • Module contents
    • Submodules
    • app.constants module
      • EnvConfig
        • EnvConfig.API_BEARER_TOKEN
        • EnvConfig.CELERY_BACKEND_CONNECTION
        • EnvConfig.CELERY_BROKER_CONNECTION
        • EnvConfig.CELERY_DEFAULT_QUEUE
        • EnvConfig.PORT
        • EnvConfig.S3_ACCESS_KEY_ID
        • EnvConfig.S3_BUCKET_NAME
        • EnvConfig.S3_ENDPOINT_URL
        • EnvConfig.S3_SECRET_ACCESS_KEY
    • app.middleware module
    • app.schemas module
      • AsyncTaskResponse
        • AsyncTaskResponse.id
        • AsyncTaskResponse.result
        • AsyncTaskResponse.status
      • FeedbackInputRequest
        • FeedbackInputRequest.income
        • FeedbackInputRequest.task_id
      • FileMetadataResponse
        • FileMetadataResponse.columns
      • UserInputRequest
        • UserInputRequest.age
        • UserInputRequest.capital_gain
        • UserInputRequest.capital_loss
        • UserInputRequest.education
        • UserInputRequest.educational_num
        • UserInputRequest.fnlwgt
        • UserInputRequest.gender
        • UserInputRequest.hours_per_week
        • UserInputRequest.marital_status
        • UserInputRequest.native_country
        • UserInputRequest.occupation
        • UserInputRequest.race
        • UserInputRequest.relationship
        • UserInputRequest.workclass
    • app.utils module
      • get_metadata()
      • retrieve_metadata()
    • Module contents