Blueprint Deployment API Reference¶
-
class
ncap_iac.utils.dev_builder.DevTemplate(filename)[source]¶ Dev mode pipelines are not hooked up to all users, and explicitly grant individuals access to a dedicated analysis bucket. Users are created, and input locations are localized to the analysis bucket in the dev case.
inputs: filename (str): the path to the stack_config_template.json blueprint that you want to deploy.
-
class
ncap_iac.utils.dev_builder.InitTemplate(filename)[source]¶ Template for initialization of a new stack. Only creates a bucket and folders, does not hook up to submit lambdas.
-
class
ncap_iac.utils.dev_builder.NeuroCaaSTemplate(filename)[source]¶ Base class for NeuroCaaS pipelines. Takes in a blueprint, and returns a viable cloudformation template that can be deployed into a pipeline. 1. There is an AWS S3 bucket that contains all input and output from the instances that we have. This is necessary because we can only have s3 buckets declared in the same template as the lambda function we will use as event sources. 2. There is an AWS Custom Resource that sets up user folders inside this bucket with input, config, submission, result and log folders. 3. There is a user group per affiliate that we have, to which we add users. All users in a given user group only have access to their folder, and furthermore only have write access to the input folder and get access from the output folder. Additional “users” can include lambda functions from other pipelines. 4. There is a lambda function that is triggered by submit files when they are passed to the input folder. We should be able to add event sources to it as we update the number of users. In addition to the standard behavior, it should set up a cloudwatch events rule to monitor given instances for state change behavior. 5. Output management. A secondary lambda function that sends notifications to an end user when processing is done. Additionally, we can use this to route output from a pipeline to another one. ##### TODO: 6. Centralized function source control. Create an AMI_updater custom resource, that takes the ami id associated with a given stack, instantiates it, pulls from the repository, and runs tests to make sure that everything is still fine. Compare with instantiating via CodeCommit/CodePipeline. The differences in this version are that we are only allowing ourselves to attach new users, not to define them.
inputs: filename (str): the path to the stack_config_template.json blueprint that you want to deploy.
-
add_affiliate(affiliatedict)[source]¶ when passed an affiliate dictionary, does two things. 1. creates the folder structure that is appropriate for this affiliate, and 2. adds a user group and users that can interact appropriately with this folder structure.
-
-
class
ncap_iac.utils.dev_builder.ReferenceUserSubstackTemplate(filename)[source]¶ Created 6/1/20 Function to create a parametrized stack that will be referenced by other pipeline stacks. Separates users associated with a given pipeline from the actual mechanics of the pipeline processing. Note that this function DOES NOT actually depend upon the specific values of the stack configuration template that is passed to it. It only uses this structure to set up a parametrized template, and at a later date the dependence on a filename should be factored out.
-
add_affiliate_parameters()[source]¶ Function to add parameters to a user subtemplate. A generator for the substack template for only one user group.
Arguments: self: (object)
The neurocaas blueprint. Should be initialized by calling the initialize_template() method.
- Outputs:
(Ref): a reference to the logical id of the folder making lambda function for the pipeline. (Ref): a reference to the physical resource id of the main analysis bucket for the pipeline. (Dict): a dictionary mocking the structure of the affiliatedictionary that is imported from the stack configuration template. Contains (Ref) objects as its entries.
-
-
class
ncap_iac.utils.dev_builder.WebDevTemplate(filename)[source]¶ Dev mode pipelines are not hooked up to all users, and explicitly grant individuals access to a dedicated analysis bucket. Input locations are localized to the analysis bucket in the dev case. NOTE 10/30: On visual inspection, it appears that the only difference between this and dev template is in the (“generate_usergroup”) function. All other functions besides initialization look to be identical to DevTemplate. This might be worth removing by first inheriting from DevTemplate instead.
inputs: filename (str): the path to the stack_config_template.json blueprint that you want to deploy.
-
class
ncap_iac.utils.dev_builder.WebSubstackTemplate(filename)[source]¶ Created 6/1/20. Newest version of the neurocaas pipeline template that makes use of substacks for every individual affiliate group.
-
add_search_lambda(output_file='end.txt')[source]¶ Add in a lambda function to do postprocessing and return output to the user. Pasted in directly from ./postprocess_lambda.py. :param output_file: the name of the file to trigger further lambda on. by default it is end.txt.
-