In this post, I will go through my experience of developing, unsuccessfully deploying, troubleshooting, and finally deploying an API to the AWS API Gateway using Zappa. In a follow-up post, I will write about my experiences of integrating an API key with this API and launching it in an API Marketplace.

Recently, I came across this interesting piece that talks about selling API as a Product. It got me excited to the point where I thought why not, let’s follow the tutorial and launch. Besides, the code has been open-sourced, so how difficult could it be? Well, it turns out, it is easy to code, but deploying it was a nightmare.

Since the original article gives an overview of writing the API and deploying it, I am writing in detail, the challenges I faced and how I overcame them.

Setting up the project

First, we need to set up a “virtual environment” and install Flask and Zappa into it

python3 -m venv env
source env/bin/activate
pip3 install flask
pip3 install zappa

Once this is done, we create an application. Create a new folder with your project name and open a new file called app.py in it. Before we actually write the code related to the article extraction API, we are going to write the basic Hello world code to test everything.

from flask import Flask
app = Flask(__name__)

@app.route('/')
def index():
    return "Hello, world!", 200

if __name__ == '__main__':
    app.run()

Simple right. Let's initialize a Zappa configuration and try to deploy this code

zappa init

This will trigger Zappa prompts that will require you to input some settings for Zappa.

███████╗ █████╗ ██████╗ ██████╗  █████╗
╚══███╔╝██╔══██╗██╔══██╗██╔══██╗██╔══██╗
  ███╔╝ ███████║██████╔╝██████╔╝███████║
 ███╔╝  ██╔══██║██╔═══╝ ██╔═══╝ ██╔══██║
███████╗██║  ██║██║     ██║     ██║  ██║
╚══════╝╚═╝  ╚═╝╚═╝     ╚═╝     ╚═╝  ╚═╝
Welcome to Zappa!
Zappa is a system for running server-less Python web applications on AWS Lambda and AWS API Gateway.This `init` command will help you create and configure your new Zappa deployment.
Let's get started!
Your Zappa configuration can support multiple production stages, like 'dev', 'staging', and 'production'.What do you want to call this environment (default 'dev'): dev
AWS Lambda and API Gateway are only available in certain regions. Let's check to make sure you have a profile set up in one that will work.Okay, using profile default!
Your Zappa deployments will need to be uploaded to a private S3 bucket.If you don't have a bucket yet, we'll create one for you too.
What do you want to call your bucket? (default 'zappa-mdhhbyymz'): article-extraction-lambda
It looks like this is a Flask application.What's the modular path to your app's function? This will likely be something like 'your_module.app'. We discovered: app.app
Where is your app's function? (default 'app.app'): app.app
You can optionally deploy to all available regions in order to provide fast global service.If you are using Zappa for the first time, you probably don't want to do this!
Would you like to deploy this application globally? (default 'n') [y/n/(p)rimary]: n
Okay, here's your zappa_settings.json:
{
  "dev": {
    "app_function": "app.app",
    "aws_region": "us-east-1",
    "profile_name": "default",
    "project_name": "articleextracti",
    "runtime": "python3.7",
    "s3_bucket": "article-extraction-lambda"
  }
}
Does this look okay? (default 'y') [y/n]: y
Done! Now you can deploy your Zappa application by executing:
$ zappa deploy dev
After that, you can update your application code with:
$ zappa update dev
To learn more, check out our project page on GitHub here: https://github.com/Miserlou/Zappa
and stop by our Slack channel here: https://slack.zappa.io
Enjoy!,
~ Team Zappa!

You will find a zappa_settings.json generated in your root directory.

Now let's deploy.

zappa deploy dev

This is where the problem started for me. If you look carefully, I have specified profile_name as default above but was trying to deploy dev. I had not created a profile dev with AWS Credentials. To do that, go to the root directory and create a folder called .aws where we will store the aws credentials for different profiles

cd 
mkdir .aws
vi .aws/credentials

Add the following credentials there

[default]
aws_access_key_id = <personal_access_key>
aws_secret_access_key = <personal_secret_access_key>
[dev]
aws_access_key_id = <project_access_key>
aws_secret_access_key = <project_secret_access_key>

Also, add a config file

vi .aws/config
[default]
region = ap-southeast-1
[dev]
region = us-east-1

These are separated into 2 files instead of keeping into 1 file so that we can keep sensitive information separate from the non-sensitive info.

Change the profile to be used in your Zappa settings to dev and let's try to re-deploy our API

zappa deploy dev

And….

The first error

Traceback (most recent call last):
File "/env/lib/python3.7/site-packages/zappa/cli.py", line 2778, in handle
sys.exit(cli.handle())
File "/env/lib/python3.7/site-packages/zappa/cli.py", line 512, in handle
self.dispatch_command(self.command, stage)
File "/env/lib/python3.7/site-packages/zappa/cli.py", line 549, in dispatch_command
self.deploy(self.vargs['zip'])
File "/env/lib/python3.7/site-packages/zappa/cli.py", line 821, in deploy
disable_progress=self.disable_progress
File "/env/lib/python3.7/site-packages/zappa/core.py", line 2226, in update_stack
raise EnvironmentError("Stack creation failed. "
OSError: Stack creation failed. Please check your CloudFormation console. You may also need to `undeploy`.

This error message did not provide much detail on its own. Upon inspecting with

zappa status
zappa tail

I could figure out that this had something to do with permissions

botocore.exceptions.ClientError: An error occurred (AccessDeniedException) when calling the GetApiKeys operation: User: arn:aws:iam:user/nagesh is not authorized to perform: apigateway:GET on resource: arn:aws:apigateway:us-east-1::/apikeys

So in the policy that Zappa created, add the required permission.

This is going to be an iterative process and you will see a lot of back and forth. But the best part about this is, Zappa will tell you what the missing permission is and you can keep on adding it.

Exception reported by AWS:An error occurred (AccessDenied) when calling the UpdateAssumeRolePolicy operation: User: arn:aws:iam:user/Article_Extraction is not authorized to perform: iam:UpdateAssumeRolePolicy on resource: role ZappaLambdaExecutionRole
botocore.exceptions.ClientError: An error occurred (AccessDeniedException) when calling the CreateFunction operation: User: arn:aws:iam:user/Article_Extraction is not authorized to perform: iam:PassRole on resource: arn:aws:iam::342716413310:role/ZappaLambdaExecutionRole

By iterating back and forth, here is what the final permission files looked like. Feel free to use this as a shortcut for your Zappa project to save you time on this back and forth.

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "iam:UpdateAssumeRolePolicy",
                "iam:AttachRolePolicy",
                "iam:CreateRole",
                "iam:GetRole",
                "iam:PutRolePolicy"
            ],
            "Resource": [
                "*"
            ]
        },
        {
            "Effect": "Allow",
            "Action": [
                "iam:PassRole"
            ],
            "Resource": [
                "arn:aws:iam::<account_id>:role/<RoleName>"
            ]
        },
        {
            "Effect": "Allow",
            "Action": [
                "apigateway:DELETE",
                "apigateway:GET",
                "apigateway:PATCH",
                "apigateway:POST",
                "apigateway:PUT",
                "events:DeleteRule",
                "events:DescribeRule",
                "events:ListRules",
                "events:ListTargetsByRule",
                "events:ListRuleNamesByTarget",
                "events:PutRule",
                "events:PutTargets",
                "events:RemoveTargets",
                "lambda:AddPermission",
                "lambda:CreateFunction",
                "lambda:DeleteFunction",
                "lambda:DeleteFunctionConcurrency",
                "lambda:GetFunction",
                "lambda:GetPolicy",
                "lambda:ListVersionsByFunction",
                "lambda:RemovePermission",
                "lambda:UpdateFunctionCode",
                "lambda:UpdateFunctionConfiguration",
                "lambda:GetAlias",
                "lambda:GetFunctionConfiguration",
                "cloudformation:CreateStack",
                "cloudformation:DeleteStack",
                "cloudformation:DescribeStackResource",
                "cloudformation:DescribeStacks",
                "cloudformation:ListStackResources",
                "cloudformation:UpdateStack",
                "logs:DescribeLogStreams",
                "logs:FilterLogEvents",
                "route53:ListHostedZones",
                "route53:ChangeResourceRecordSets",
                "route53:GetHostedZone",
                "s3:CreateBucket"
            ],
            "Resource": [
                "*"
            ]
        },
        {
            "Effect": "Allow",
            "Action": [
                "s3:ListBucket"
            ],
            "Resource": [
                "arn:aws:s3:::<s3_bucket_name>"
            ]
        },
        {
            "Effect": "Allow",
            "Action": [
                "s3:DeleteObject",
                "s3:GetObject",
                "s3:PutObject",
                "s3:CreateMultipartUpload",
                "s3:AbortMultipartUpload",
                "s3:ListMultipartUploadParts",
                "s3:ListBucketMultipartUploads"
            ],
            "Resource": [
                "arn:aws:s3:::<s3_bucket_name>/*"
            ]
        }
    ]
}

With all the permissions provided, I was all ready to see my API live, when suddenly

Packaging project as zip.
Uploading articleextracti-dev-1588309879.zip (6.0MiB)..
100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 6.25M/6.25M [00:14<00:00, 424kB/s]
Scheduling..
Scheduled articleextracti-dev-zappa-keep-warm-handler.keep_warm_callback with expression rate(4 minutes)!
Uploading articleextracti-dev-template-1588309904.json (1.6KiB)..
100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 1.60k/1.60k [00:00<00:00, 1.96kB/s]
Waiting for stack articleextracti-dev to create (this can take a bit)..
75%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████                                            | 3/4 [00:14<00:04,  4.83s/res]
Deploying API Gateway..
Error: Warning! Status check on the deployed lambda failed. A GET request to '/' yielded a 502 response code.

The code was running locally, yet I was not sure, why it was throwing a 500 error. This was probably one of the hardest parts to debug. I tried to read a lot of blogs, answers but they were of not much help. I made a lot of tweaks to the code, removed any dependencies, and did a lot of deploy un-deploy cycles, but to no real gain. Once I removed all unnecessary packages, I added them one by one, as needed in requirements.txt file

Once all the requirements were clean, I realized one of the major reason for the 5xx errors for Zappa are incompatible libraries and python versions on AWS. Here is my final updated Zappa settings file

{
    "dev": {
        "app_function": "app.app",
        "aws_region": "us-east-1",
        "profile_name": "dev",
        "project_name": "articleextracti",
        "runtime": "python3.6",
        "s3_bucket": "nagesh-article-extraction-lambda",
        "role_name": "ZappaLambdaExecutionRole"
    }
}

I downgraded the python version to 3.6. Doing a deploy now with the basic python code worked.

Your updated Zappa deployment is live!: https://p4jtzmxejk.execute-api.us-east-1.amazonaws.com/dev

From there on, I incrementally added packages and wrote the article extraction code. I updated the deployment frequently to check things were working fine.

zappa update dev

In the next part, I will share my experiences about How I integrated an API key and launched it on the API Marketplace. Stay tuned.