gable data-asset check - Gable Docs

Documentation Index

Fetch the complete documentation index at: /llms.txt

Use this file to discover all available pages before exploring further.

Checks data asset(s) against a contract

Usage

gable data-asset check [OPTIONS]

Options

Option Type Required Default Description
--source-type postgres mysql mssql json_schema
--include-unchanged-assets boolean false Include assets that are the same as Gable’s registered version of the asset. This is useful for checking the current state of an asset but should not be used for automated checks on branches.
-o, --output text json markdown
--host, -h text The host name of the production database, for example ‘service-one.xxxxxxxxxxxx.us-east-1.rds.amazonaws.com’. Despite not needing to connect to the production database, the host is still needed to generate the unique resource name for the real database tables (data assets).
--port, -p integer The port of the production database. Despite not needing to connect to the production database, the port is still needed to generate the unique resource name for the real database tables (data assets).
--db text The name of the production database. Despite not needing to connect to the production database, the database name is still needed to generate the unique resource name for the real database tables (data assets). Database naming convention frequently includes the environment (production/development/test/staging) in the database name, so this value may not match the name of the database in the proxy database instance. If this is the case, you can set the —proxy-db value to the name of the database in the proxy instance, but we’ll use the value of —db to generate the unique resource name for the data asset. For example, if your production database is ‘prod_service_one’, but your test database is ‘test_service_one’, you would set —db to ‘prod_service_one’ and —proxy-db to ‘test_service_one’.
--schema, -s text The schema of the production database containing the table(s) to check. Despite not needing to connect to the production database, the schema is still needed to generate the unique resource name for the real database tables (data assets). Database naming convention frequently includes the environment (production/development/test/staging) in the schema name, so this value may not match the name of the schema in the proxy database instance. If this is the case, you can set the —proxy-schema value to the name of the schema in the proxy instance, but we’ll use the value of —schema to generate the unique resource name for the data asset. For example, if your production schema is ‘production’, but your test database is ‘test’, you would set —schema to ‘production’ and —proxy-schema to ‘test’.
--table, --tables, -t text A comma delimited list of the table(s) to check. If no table(s) are specified, all tables within the provided schema will be checked. Table names in the proxy database instance must match the table names in the production database instance, even if the database or schema names are different.
--proxy-host, -ph text The host string of the database instance that serves as the proxy for the production database. This is the database that Gable will connect to when checking tables in the CI/CD workflow.
--proxy-port, -pp integer The port of the database instance that serves as the proxy for the production database. This is the database that Gable will connect to when checking tables in the CI/CD workflow.
--proxy-db, -pdb text Only needed if the name of the database in the proxy instance is different than the name of the production database. If not specified, the value of —db will be used to generate the unique resource name for the data asset. For example, if your production database is ‘prod_service_one’, but your test database is ‘test_service_one’, you would set —db to ‘prod_service_one’ and —proxy-db to ‘test_service_one’.
--proxy-schema, -ps text Only needed if the name of the schema in the proxy instance is different than the name of the schema in the production database. If not specified, the value of —schema will be used to generate the unique resource name for the data asset. For example, if your production schema is ‘production’, but your test database is ‘test’, you would set —schema to ‘production’ and —proxy-schema to ‘test’.
--proxy-user, -pu text The user that will be used to connect to the proxy database instance that serves as the proxy for the production database. This is the database that Gable will connect to when checking tables in the CI/CD workflow.
--proxy-password, -ppw text If specified, the password that will be used to connect to the proxy database instance that serves as the proxy for the production database. This is the database that Gable will connect to when checking tables in the CI/CD workflow.
--files tuple Space delimited path(s) to the assets to check, with support for glob patterns.
--project-root text This should be the directory location of the Python project that will be analyzed.
--emitter-function text Name of the emitter function
--emitter-payload-parameter text Name of the parameter representing the event payload
--event-name-key text Input must be a ”.” delimited list of field property access directives or array indexes to the event name key field. The field property access directive is a valid dictionary key, or the wildcard character ".". The array index is square brackets[] with either a digit in it (targeting a specific element of the array), or * (targeting all elements in the array). Example: “fieldName.[0].eventName” describes the access pay to the event name in: {'fieldName': [{'eventName': 'event_one'}]}.
--emitter-file-path text Relative path from the root of the project to the file that contains the emitter function
--exclude text Comma separated list of paths to be excluded from the analysis, with support for glob patterns. Gable automatically excludes '**/node_modules, **/pycache, /.*'. Example: '/tests,docs/*'
--project-root text This should be the directory location of the Typescript project that will be analyzed.
--library brandviews segment amplitude udf
--rules-file text File containing match rules for egress points. Can be used in conjunction with —library, but takes precedence over —emitter-* args.
--node-modules-include text Comma delimited list of filenames or patterns of node modules to include in the analysis.
--emitter-file-path text DEPRECATED: Use —emitter-location instead.
--emitter-location text NPM package name, or relative path from the root of the project to the file that contains the emitter function
--emitter-function text Name of the emitter function. This can be a standalone function like ‘trackEvent’ or a class method like ‘AnalyticsClient.track’
--emitter-payload-parameter text Name of the parameter representing the event payload
--emitter-name-parameter text Name of the emitter function parameter that contains the event name. Either this option, or the —event-name-key option must be provided when using —emitter-function.
--event-name-key text Name of the event property that contains the event name. Either this option, or the —emitter-name-parameter option must be provided when using —emitter-function.
--exclude text Comma delimited list of filenames or extended globbing patterns of node modules to include in the analysis. Defaults to exclude common test patterns like *.test.js, *.spec.js, etc.
--project-root text This should be the directory location of the project containing the Pyspark job that will analyzed.
--spark-job-entrypoint text Entrypoint to execute spark job, starting with python file and including any arguments. Example: “main.py —arg1 value1 —arg2 value2”
--connection-string text Connection string to Hive cluster used to pull schemas of input tables
--metastore-connection-string text Connection string to the Hive metastore used to pull S3 to table mappings
--csv-schema-file text Path to the CSV schema file
--csv-path-to-table-file text Path to a CSV schema file containing a mapping of Delta table paths to table names
--config-file file Path to YAML configuration file containing the necessary configurations for the Pyspark job.
--config-entrypoint-path text The path to the property of the YAML config file containing the spark job entrypoint, which is the main Python script for the job. For example: ‘spec.mainApplicationFile’
--config-args-path text The path to the property of the YAML config file containing the spark job arguments. For example: ‘spec.arguments’
--bucket text This should indicate the S3 bucket containing the files to be analyzed.
--include-prefix text This optional parameter allows you to specify what to include in your S3 bucket. If not specified, all files in the bucket will be analyzed.
--exclude-prefix text This optional parameter allows you to specify what to exclude in your S3 bucket. If —include-prefix is specified, this parameter must be a subset of include to be considered.
--lookback-days integer 2 Number of days to look back from the latest day in the list of paths, defaults to 2. For example if the latest path is 2024/01/02, and lookback_days is 3, then the paths return will have 2024/01/02, 2024/01/01, and 2023/12/31
--history boolean false This optional parameter allows you to do a historical analysis between 2 dates.
--skip-profiling boolean false This optional parameter allows you to turn off data profiling.
--row-sample-count integer 1000 Number of rows of data per file to sample for schema detection and data profiling. Default is 1000. Accuracy increases with larger sample size, but processing time and AWS costs also increases.
--recent-file-count integer (≥1) 3 Specifies the number of most recent files whose schema will be used for inference per data asset. Default is 3. For example, if the latest file is 2024/01/10 and --recent-file-count is 2, then only files 2024/01/10 and 2024/01/09 will be used for schema inference, even if —lookback-days is greater than 2. Increase this value to improve schema accuracy over more schema history, at the cost of increased runtime. Must be at least 1.
--endpoint text Customer API endpoint for Gable, in the format https://api.company.gable.ai/. Can also be set with the GABLE_API_ENDPOINT environment variable.
--api-key text API Key for Gable. Can also be set with the GABLE_API_KEY environment variable.

Example

gable data-asset check --source-type protobuf --files ./**/*.proto