SuperAPI AI Directory
Partner with us中文AI newsFavorites
Skillsbyted-emr-skills

byted-emr-skills

Data & analytics

View repository

Updated 10-15 17:30

About this skill

EMR Skills
  • When to use (trigger phrases)
  • When the user raises any of the following similar requests, invoke this skill immediately: 
    “查询EMR集群的服务列表”
“重启Spark服务”
“获取 EMR 作业日志”
“查看EMR serverless队列列表”
“查看EMR serverless队列详情”
“使用 EMR Serverless 运行任务”
“在 EMR 上提交一个 Spark 作业”
“帮我诊断一下失败的 Spark 作业”
“分析一下EMR作业失败的原因”
  • Any operational inquiry involving EMR Serverless jobs/queues/compute groups/monitoring/logs/diagnostics
Feature list
  • EMR Serverless
    • Queues: create public queue, queue list query, queue details query, compute group list query under a queue
    • Queue permissions: query queue authorization principals, grant/modify queue permissions for users
    • Compute groups: create/modify/start/stop/delete, query available specifications and images, external metadata connectivity test
    • Job templates: create/update/query details/list query/run
    • Job instances: submit (SparkSQL/PrestoSQL/SparkJar/PySpark/RayJob), query status and details, cancel, list query, rerun, pull results in batches
  • EMR Agent
    • Interactive diagnostics and knowledge Q&A
    • Session and report management
  • EMR on ECS
    • Clusters: view cluster details, cluster list, update cluster attributes
    • Node groups: list node groups, update node group attributes, node list, scale up node group disks, update node group ECS specifications
    • Services: list applications, component list, component instance list, execute application operations
    • Service configuration: configuration file list, configuration file details, configuration item list, configuration item modification history list, modify configuration items
    • Operation audit: operation list query
    • Users: create cluster users, modify created user information, get cluster user list, get user details, update cluster user passwords
    • User groups: get cluster user group list, cluster user group details, create cluster user groups, update cluster user groups
  • EMR on VKE
    • Clusters: view cluster list, cluster details
    • Services (applications): view service list, service component instances, restart services or component instances, view service configuration parameters, modify service configuration parameters
    • Operation logs: view operation log list, operation log details
Initial Setup

Environment variable configuration

Method 1: Use the ARK_SKILL_API environment variable (recommended)

If the following environment variables are already configured, there is no need to configure Volcengine API credentials:

export ARK_SKILL_API_BASE="your-api-base-url"
export ARK_SKILL_API_KEY="your-api-key"

Method 2: Use Volcengine API credentials

If the ARK_SKILL_API environment variable is not configured and you need to use serverless SDK capabilities or an SDK error occurs, you need to configure Volcengine API credentials:

export VOLCENGINE_AK="your-access-key"
export VOLCENGINE_SK="your-secret-key"
export VOLCENGINE_REGION="cn-beijing"
  • Install the emr serverless SDK by calling scripts/bin/install_serverless_sdk.sh to install
How to manage EMR

1. EMR on Serverless management

  • For the fully managed Serverless form, it mainly targets job submission and resource queue management.
  • OpenAPI recommends using the command-line tool:python ./scripts/on_serverless/emr_serverless_cli.py --action <Action> --method <GET|POST> --query '<json>' --body '<json>'
    • Region is read from the environment variable by default VOLCENGINE_REGION (default cn-beijing)
    • Service/Version will be automatically inferred based on the Action (it can also be explicitly passed when needed --service/--version）
    • A custom endpoint (such as LAS) can additionally pass --endpoint las.cn-beijing.volcengineapi.com
  • Job submission recommends using the command-line tool:python ./scripts/on_serverless/emr_serverless_submit_cli.py <sql|jar|pyspark|ray> ...

How to manage resource queues

  • List of supported features
    • Queue list: view all resource queues and their configurations (such as maximum resources and current usage); OpenAPI must be usedListTagQueueQuery the queue list.
    • Queue details: obtain the detailed information of a queue, including bound networks, storage, etc.; OpenAPI must be usedGetQueueQuery queue details.
    • Create queue: create a public queue with one click; OpenAPI must be usedCreateQueueSilentlyorCreateOrderInOneStepCreate a queue.
    • Queue component list: obtain the list and details of the queue components under a specified queue; OpenAPI must be usedListQueueComponentQuery the queue component list.
  • For all details of queue feature operations, please strictly followreferences/emr_serverless/queue/emr_serverless_queue_guide.mdthe instructions in

How to manage queue permissions

  • List of supported features
    • Get queue permission list: based on a specified queue, obtain the list of users/user groups that have its permissions; OpenAPI must be usedListAuthorizedPrincipalsForQueueQuery the queue permission list.
    • Fuzzy search queue permissions: perform a fuzzy query of the user list based on username, and at the same time determine whether the user already has permissions for the specified queue; OpenAPI must be usedListIAMUsersWithQueueRoleQuery the queue permission list.
    • Add user permission: grant a user permission for a specified data object; OpenAPI must be usedGrantQueuePrivilegeAdd user permission.
    • Modify user queue permissions: modify a user's permissions for a specified queue; OpenAPI must be usedAlterQueuePrivilegeModify user permissions.
  • For all details of feature operations, please strictly followreferences/emr_serverless/privilege/emr_serverless_privilege_guide.mdthe instructions in

How to manage job templates

  • Note the difference between job templates and job instances:
    • Job template: defines the general configuration of a job, including code path, parameters, environment variables, etc.
    • Job instance: a specific running instance created based on a job template, including job ID, runtime parameters, status, etc.
  • List of supported job template features
    • Create job template: define a job template (including code path, parameters, environment configuration, etc.); OpenAPI must be usedCreateJobDefinitionCreate a job template.
    • Execute or run job template: trigger a run based on an existing job template; OpenAPI must be usedRunJobDefinitionExecute a job template.
    • Modify job template: modify a job template (such as execution resources, code path, entry command, etc.), but it does not affect submitted job instances; OpenAPI must be usedAlterJobDefinitionModify a job template.
    • Query job template details: obtain the detailed configuration information of a job template; OpenAPI must be usedGetJobDefinitionQuery job template details.
    • Query job template list: obtain the list of all created job templates; OpenAPI must be usedListJobDefinitionsQuery the job template list.
  • For the detailed operation guide for all job templates (that is, job definitions), please strictly followreferences/emr_serverless/job/emr_serverless_job_guide.mdthe instructions in

How to manage job instances

  • List of job instance features
    • Submit jobs: supports types such as SparkSQL, PrestoSQL, SparkJar, PySpark, and RayJob; SparkJar/PySpark/RayJob support job execution scripts using local paths (the SDK automatically uploads them to TOS; for details, see the job instance guide).
    • Query job instances: it is recommended to use OpenAPI QueryGetJobV2 to get status and details.
    • Terminate job instances: OpenAPI QueryCancelQueryV2。
    • Retrieve result data in batches: for SQL-type jobs, you can use OpenAPI QueryFetchResultsByBatch to retrieve results in batches
    • Job instance list: OpenAPI ListJobInstances。
  • For detailed operation guides for all job instances, please strictly followreferences/emr_serverless/job_instance/emr_serverless_job_instance_guide.mdinstructions in.

How to manage compute groups (Serverless resource units)

  • Create/modify compute group configurations.

How to query operation logs

  • Feature list
    • Query the global operation log page and the operation lists of queues and compute groups
  • For all operation log queries, please strictly followreferences/emr_serverless/operation_audit/emr_serverless_operation_audit_guide.mdinstructions in.

2. EMR Agent (EMR AI Assistant)

How to perform interactive problem diagnosis and knowledge assistance

  • Feature list
    • Applicable to all EMR forms, providing interactive problem diagnosis and knowledge Q&A.
  • Billing description
    • EMR Agent on the EMR console currently requires whitelisting; API calls do not currently require whitelisting and can be used directly
    • There is currently a certain free quota, and billing will be based on usage later.

How to manage diagnostic sessions and report management

  • Feature list
    • Report list: get the list of historical diagnostic reports.
    • View report details: get the content of an existing report.
    • Session list: get the list of historical sessions for analysis, diagnosis, and Q&A.
    • View session details: get detailed information about a session, including problem description, answer content, session status, etc.
  • For all analysis and diagnosis sessions and report operations, please strictly followreferences/emr_agent/emr_agent_guide.mdinstructions in.

3. EMR on ECS cluster management

  • Supports full lifecycle management of traditional Hadoop clusters based on ECS.

How to manage clusters

  • Feature list
    • View cluster details: get basic cluster information, status, billing type, access address, etc.; you must use OpenAPIGetClusterQuery cluster details.
    • Cluster list: filter by conditions and paginate queries; you must use OpenAPIListClustersQuery cluster list.
    • Update cluster attributes: modify cluster name, description, etc.; you must use OpenAPIUpdateClusterAttributeUpdate cluster attributes.
  • For details of all cluster management operations, please strictly followreferences/emr_on_ecs/cluster/emr_on_ecs_cluster_guide.mdinstructions in.

Node group management

  • Feature list
    • List node groups: view the node groups in the cluster (such as Master, Core, Task) and their configurations; you must use OpenAPIListNodeGroupsList the node group list.
    • Update node group attributes: modify node group name, instance specifications (upgrades supported), disk scaling; you must use OpenAPIUpdateNodeGroupAttributeUpdate node group attributes.
    • Node list: view detailed information of all ECS instances in a node group; you must use OpenAPIListNodesQuery detailed information of all node instances in the node group.
    • Scale up node group disk: scale up the disk capacity of the specified node group. OpenAPI must be used.ResizeNodeGroupDiskScale up node group disk.
    • Update node group ECS specification: modify the ECS instance specification of the specified node group. OpenAPI must be used.UpdateNodeGroupEcsSpecUpdate node group ECS specification.
  • For details of all node group management operations, strictly follow the instructions inreferences/emr_on_ecs/node_group/emr_on_ecs_node_group_guide.mdto perform the operations.

Application Management

  • Feature List
    • List applications: view the big data components installed in the cluster. OpenAPI must be used.ListApplicationsList the application list.
    • Component list: view the role distribution of each service. OpenAPI must be used.ListComponentsList the component list.
    • Component instance list: view the instance list of each component. OpenAPI must be used.ListComponentInstancesList the instance list of component instances.
    • Execute application operations: perform operations such as start, stop, restart, and rolling restart on components. OpenAPI must be used.RunApplicationActionExecute application operations.
  • For details of all application management operations, strictly follow the instructions inreferences/emr_on_ecs/application/emr_on_ecs_application_guide.mdto perform the operations.

Application Configuration Management

  • Feature List
    • Configuration file list: view the configuration file list of the specified application in the cluster. OpenAPI must be used.ListApplicationConfigFilesList the application configuration file list.
    • Configuration file details: view information such as the content and modification time of the configuration file of the specified application in the cluster. OpenAPI must be used.GetApplicationConfigFileQuery application configuration file details.
    • Configuration item list: view the configuration item list of the specified application in the cluster, including information such as configuration item name, current value, and default value. OpenAPI must be used.ListApplicationConfigsList the application configuration item list.
    • Configuration item modification history list: view the configuration item modification history list of the specified application in the cluster. OpenAPI must be used.ListApplicationConfigHistoriesList the application configuration item modification history list.
    • Modify configuration item: modify the configuration item value of the specified application in the cluster. OpenAPI must be used.UpdateApplicationConfigUpdate application configuration items.
  • For details of all application configuration management operations, strictly follow the instructions inreferences/emr_on_ecs/application_config/emr_on_ecs_application_config_guide.mdto perform the operations.

Operation Audit Management

  • Operation list query: view all operation logs in the cluster, with support for paginated queries. OpenAPI must be used.ListOperationsList the operation list.
  • For details of all operation audit management operations, strictly follow the instructions inreferences/emr_on_ecs/operation/emr_on_ecs_operation_guide.mdto perform the operations.

User Management

  • Feature List
    • Create cluster user: add a user to the cluster, requiring information such as username and password. OpenAPI must be used.CreateClusterUserCreate a cluster user.
    • Modify created user information: modify the user group or description information of an existing user in the specified cluster. OpenAPI must be used.UpdateClusterUserUpdate cluster user information.
    • Get cluster user list: query the user information list of the specified cluster. OpenAPI must be used.ListClusterUsersList the cluster user list.
    • Get user details: query detailed user information of the specified cluster. OpenAPI must be used.GetClusterUsersQuery cluster user details.
    • Update cluster user password: Modify the login password of an existing user in the specified cluster. OpenAPI must be used.UpdateClusterUserPasswordUpdate cluster user password.
  • For details of all user management operations, please strictly followreferences/emr_on_ecs/user/emr_on_ecs_user_guide.mdto perform the operations.

User Group Management

  • Feature List
    • Get cluster user group list: Query the user group information list of the specified cluster. OpenAPI must be used.ListClusterUserGroupsList the cluster user group list.
    • Cluster user group details: Query the detailed user group information of the specified cluster. OpenAPI must be used.GetClusterUserGroupsQuery cluster user group details.
    • Create cluster user group: Create a new user group for the specified cluster. OpenAPI must be used.CreateClusterUserGroupCreate cluster user group.
    • Update cluster user group: Modify the description or member information of an existing user group in the specified cluster. OpenAPI must be used.UpdateClusterUserGroupUpdate cluster user group.
  • For details of all user group management operations, please strictly followreferences/emr_on_ecs/user_group/emr_on_ecs_user_group_guide.mdto perform the operations.

EMR on VKE Cluster Management

  • For details of all cluster management operations, please strictly followreferences/emr_on_vke/emr_on_vke_guide.mdto perform the operations.
Available Scripts
  • scripts/on_serverless/emr_serverless_manager.py: Unified EMR Serverless OpenAPI call entry point (manage_emr_serverless), used for queue/job instance/compute group and other O&M-type API calls.
  • scripts/config/config.py: Encapsulates the configurations used by EMR, SDK Client construction, etc.
  • scripts/emr_agent/expert.py: Interacts with EMR Agent to implement intelligent diagnosis and knowledge Q&A.
  • scripts/emr_agent/emr_agent_manager.py: Get the EMR Agent's session list, report list, etc.
  • scripts/bin/install_serverless_sdk.sh: Install the EMR on Serverless form Python SDK, used for Serverless queue management, job submission, and other operations.
  • scripts/on_ecs/emr_on_ecs_manager.py: Unified EMR on ECS form OpenAPI call entry point (manage_emr_on_ecs), used for cluster/node group/application and other O&M-type API calls.
  • scripts/on_ecs/emr_on_ecs_cli.py: Command-line tool for the EMR on ECS form, used for cluster/node group/application and other O&M-type API calls.
  • scripts/on_vke/emr_on_vke_manager.py: Management script for the EMR on VKE form product
References
  • references/emr_serverless/queue/emr_serverless_queue_guide.md: Detailed management operation guide for resource queues in the EMR on Serverless form
  • references/emr_serverless/job_instance/emr_serverless_job_instance_guide.md: Detailed management operation guide for job instances in the EMR on Serverless form
  • references/emr_serverless/job/emr_serverless_job_guide.md: Management operation guide for job templates (i.e., job definitions) in the EMR on Serverless form
  • references/emr_serverless/compute/emr_serverless_compute_guide.md: Detailed management operation guide for compute groups in the EMR on Serverless form
  • references/emr_agent/emr_agent_guide.md: Detailed operation guide for EMR Agent, including problem analysis, obtaining historical sessions, obtaining historical diagnostic reports, etc.
  • references/emr_on_ecs/cluster/emr_on_ecs_cluster_guide.md: Operation guide for EMR on ECS cluster management, including cluster list query, cluster details query, updating cluster attributes, etc.
  • references/emr_on_ecs/node_group/emr_on_ecs_node_group_guide.md: Operation guide for EMR on ECS node group management, including node group list query, node group details query, updating node group attributes, etc.
  • references/emr_on_ecs/application/emr_on_ecs_application_guide.md: Operation guide for EMR on ECS application management, including application list query, component list query, component instance list query, executing application operations, etc.
  • references/emr_on_ecs/application_config/emr_on_ecs_application_config_guide.md: Operation guide for EMR on ECS application configuration management, including configuration file list query, configuration file details query, configuration item list query, configuration item modification history list query, modifying configuration items, etc.
  • references/emr_on_ecs/operation/emr_on_ecs_operation_guide.md: Operation guide for EMR on ECS operation audit management, including operation list query, etc.
  • references/emr_on_ecs/user/emr_on_ecs_user_guide.md: EMR on ECS user management operation guide, including creating users, modifying user information, querying the user list, querying user details, updating user passwords, etc.
  • references/emr_on_ecs/group/emr_on_ecs_group_guide.md: EMR on ECS user group management operation guide, including querying the user group list, querying user group details, creating user groups, updating user groups, etc.
  • references/emr_on_vke/emr_on_vke_guide.md: EMR on VKE cluster management operation guide. Including querying the cluster list and details, querying the service list and details, querying the operation log list and details, restarting services, querying and updating service configuration parameters, etc.
Assets
  • assets/libs/python_serverless-1.*-py3-none-any.whl: The Python SDK package for the EMR on Serverless form, used for operations such as EMR Serverless job submission.
  • assets/emr-serverless-job-template.json: Serverless job submission template, including PySpark job parameter examples.
Requirements
  • If temporary files need to be written (such as writing scripts, keeping memos, etc.), the files must be placed in the ./tmp folder.
  • When encountering job, service, or cluster failures, or when the user has diagnostic needs, you must prioritize using the EMR Agent's diagnostic function rather than analyzing and diagnosing on your own. The EMR Agent diagnosis takes a certain amount of time, so you should wait for it to complete and not kill the diagnostic task midway.
  • Do not read the contents of the scripts under scripts; just execute according to the instructions in the guide document, unless you have a necessary reason to read them (for example, frequent execution failures require troubleshooting, or the user asks about implementation details, etc.).
  • When you import packages under script through Python code, you need to add the root directory of emr-skills to path: sys.path.append(${emr_skill_root})
  • When you submit a task through Python code and wait for completion, you need to split it into two Python scripts: first execute the task submission script, and after obtaining the job_id, then execute the status retrieval script; the status retrieval script should not use a While loop inside the script to keep retrieving the status, but should retrieve the status only once, and continuously retrieve the status by running the script multiple times.