\n $\"components\"$ \n

\n:information_source: Some environment variables with opinionated naming conventions do not necessarily imply that the stacks are bound to a specific\ncloud provider or even using a cloud service at all. For example, `AWS_ACCESS_KEY_ID` and `AWS_SECRET_ACCESS_KEY` are used\nto set credentials for `minio`, the default storage service, rather than for Amazon Web Services.\n\n#### Model lifecycle management\nCMS has embedded a `mlflow` stack to provide a pipeline of model lifecycle management, covering tracking of training/fine-tuning,\nmodel registration, versioning on new models and so on.\n```commandline\nexport MLFLOW_DB_USERNAME=\nexport MLFLOW_DB_PASSWORD=\nexport AWS_ACCESS_KEY_ID=\nexport AWS_SECRET_ACCESS_KEY=\ndocker compose -f docker-compose-mlflow.yml up -d\n```\n\nThe user authentication feature is optional and turned off by default. If you wish to enable it, consult the [Management](./app/management/README.md)\ndocumentation. The newly trained model and the metrics collected during training will be stored in the `minio` bucket and\nthe `mlflow` database, rather than the local file system.\n\n#### Work with the local/remote model registry:\nAll models are registered in a so-called model registry, which can be hosted either on a local machine or on a remote HTTP server. To work with the model registry, the prerequisites include setting up the following environment variables:\n```commandline\nexport AWS_ACCESS_KEY_ID=\nexport AWS_SECRET_ACCESS_KEY=\nexport MLFLOW_TRACKING_URI=\n```\n\nAfter that, you can run model serving with a registered model. For example, with `minio` as the model storage backend,\n```commandline\ncms serve --model-type medcat_icd10 --mlflow-model-uri s3://cms-model-bucket/EXPERIMENT_ID/RUN_ID/artifacts/REGISTERED_MODEL_NAME --host 0.0.0.0 --port 8000\n```\n\nYou may want to push the package of a pretrained model to the registry before serving. For example, with `minio` being the model storage backend,\n```commandline\ncms register --model-type medcat_snomed --model-path PATH/TO/MODEL_PACKAGE.zip --model-name REGISTERED_MODEL_NAME\n```\n\nYou can also serve a registered model through standard MLflow APIs instead of ModelServe APIs:\n```commandline\nmlflow models serve -m s3://cms-model-bucket/EXPERIMENT_ID/RUN_ID/artifacts/REGISTERED_MODEL_NAME -h 0.0.0.0 -p 8001\n```\nThen the `/invocations` endpoint will be up and running for model scoring. For example:\n```commandline\ncurl http://127.0.0.1:8001/invocations \\\n -H 'Content-Type: application/json' \\\n -d '{\"dataframe_split\": { \"columns\": [\"name\", \"text\"], \"data\": [[\"DOC\", \"TEXT\"], [\"ANOTHER_DOC\", \"ANOTHER_TEXT\"]]}}'\n```\n\n#### Monitoring and observability\nBy default, CMS exposes most services via HTTP APIs. Their monitoring can be performed by running the `mon` stack:\n```commandline\nexport GRAFANA_ADMIN_USER=\nexport GRAFANA_ADMIN_PASSWORD=\ndocker compose -f docker-compose-mon.yml up -d\n```\n\n#### Log management\nCMS provides a `log` stack to run services used for centralised logging and log analysis:\n```commandline\nexport GRAYLOG_PASSWORD_SECRET=`openssl rand -hex 32`\nexport GRAYLOG_ROOT_PASSWORD_SHA2=`echo -m \"CHANGE_ME\" | sha256`\ndocker compose -f docker-compose-log.yml up -d\n```\n\nAfter running this stack for the first time, you will need to create a default input to which CMS can send logs:\n\n```commandline\ncurl -u \"admin:CHANGE_ME\" \\\n-X POST \"http://localhost:9000/api/system/inputs\" \\\n-H \"Content-Type: application/json\" \\\n-H \"X-Requested-By: curl\" \\\n-d '{\n \"title\": \"CMS GELF Input\",\n \"type\": \"org.graylog.plugins.gelf.inputs.GELFTCPInput\",\n \"global\": true,\n \"configuration\": {\n \"bind_address\": \"0.0.0.0\",\n \"port\": 12201\n }\n}'\n```\n\n#### Proxying\nA reverse proxying service is available inside the `proxy` stack:\n```commandline\ndocker compose -f docker-compose-proxy.yml up -d\n```\n\n#### API Authentication\nAn optional user database can be spun up if authentication is needed:\n```commandline\nexport AUTH_DB_USERNAME=\nexport AUTH_DB_PASSWORD=\ndocker compose -f docker-compose-auth.yml up -d\n```\nBefore running ModelServe, define extra environment variables to enable the built-in\ntoken-based authentication and hook it up with the database by following this [Instruction](./app/api/auth/README.md).\n\n### Serve models via streaming HTTP APIs\nYou can send your texts to the CMS stream endpoint and receive NLP results as a stream. To that end,\nstart CMS as a streamable service by running:\n```commandline\ncms serve --streamable --model-type --model-path PATH/TO/MODEL_PACKAGE.zip --host 127.0.0.1 --port 8000\n```\nCurrently, [JSON Lines](https://jsonlines.org/) is supported for formatting request and response bodies. For example, the following request:\n```commandline\ncurl -X 'POST' 'http://127.0.0.1:8000/stream/process' \\\n -H 'Content-Type: application/x-ndjson' \\\n --data-binary $'{\"name\": \"DOC\", \"text\": \"TEXT\"}\\n{\"name\": \"ANOTHER_DOC\", \"text\": \"ANOTHER_TEXT\"}'\n```\nwill result in a response like {\"doc_name\": \"DOC\", \"start\": INT, \"end\": INT, \"label_name\": \"STR\", \"label_id\": \"STR\", ...}\\n...\n\n### Serve causal language models\nYou can serve causal LLMs (e.g., LLaMa 3 or DeepSeek R1) using the `huggingface_llm` model type. To do so, run:\n```commandline\ncms serve --model-type huggingface_llm --model-path PATH/TO/MODEL_PACKAGE.zip --host 127.0.0.1 --port 8000\n```\nTo generate texts in near real-time, send a request with the optional query parameter `max_tokens`:\n```commandline\ncurl -N -X 'POST' 'http://127.0.0.1:8000/stream/generate?max_tokens=512' \\\n -H 'accept: application/json' -H 'Content-Type: text/plain' \\\n -d 'What is hypertension?'\n```\n\n#### Chat with served models\nYou can also \"chat\" with the running model using the `/stream/ws` endpoint. For example:\n```html\n\n

\n\n```","has_starred","hub_user","immutable_tags_settings",{"_140":11,"_141":142},"enabled","rules",[143],".*","is_automated","permissions",{"_147":22,"_148":11,"_149":11},"read","write","admin","user","storage_size",519480419087,"isOrg","reponame","tag","canDeleteTags","isHubSbxKitsEnabled","currentTab","tagAttestations","displaySecurityMetadata","showSbxLogoOverride","routes/_layout.layers.$namespace.$reponame.$tag.(images).($imageUrlDigest)._index",{"_28":29,"_102":-7,"_100":-7,"_101":-7,"_30":164,"_36":47,"_170":171,"_172":11},{"_32":33,"_34":35,"_36":37,"_38":39,"_40":41,"_42":165,"_79":80,"_81":35,"_82":83,"_84":85,"_86":87,"_88":89,"_90":71,"_91":80,"_92":60,"_93":22},[166,167,168,169],{"_45":46,"_36":47,"_48":49,"_50":51,"_52":53,"_54":55,"_56":49,"_57":-5,"_58":39,"_59":60,"_61":-5},{"_45":63,"_36":64,"_48":49,"_50":65,"_52":66,"_54":55,"_56":49,"_57":-5,"_58":67,"_59":60,"_61":-5},{"_45":69,"_36":70,"_48":49,"_50":71,"_52":72,"_54":69,"_56":49,"_57":-5,"_58":73,"_59":60,"_61":-5},{"_45":69,"_36":75,"_48":49,"_50":76,"_52":77,"_54":69,"_56":49,"_57":-5,"_58":78,"_59":60,"_61":-5},"imageWithLayers",{"_45":46,"_36":47,"_48":49,"_50":51,"_52":53,"_54":55,"_56":49,"_57":-5,"_58":39,"_59":60,"_61":-5},"hasPermissionToViewRepoSettings","actionData","errors"]

cogstacksystems/cogstack-modelserve:pr-45