Module 4: Working With Other AWS Services
LEARNING MODULE
Please note, you may follow along with the examples presented here, but you do not need to.
There are a few ways to work with other AWS services.
If the service you are accessing is an AWS RDS Database, like SQL Server, or Postgres, you use the same libraries you would use if you hosted the databases on your own computer or data center. You need a connection string with username and password, or another form of authentication of your choice. There is nothing different from your day-to-day usage of the database. You don't need any additional permissions to access the database server. The only caveat to this is if the database is not publicly accessible, then you need to connect the Lambda to the VPC (that process requires extra permissions).
If your Lambda function is using S3, DynamoDB, Kinesis, etc, you use the AWS SDKs to interact with those services. The role your Lambda function is running under needs appropriate permissions to interact with each service. For example, if you want to add an item to an S3 bucket, the role will need permission to write to that bucket. If you want to get items from a DynamoDB table, the role will need permissions to read from that table.
The third scenario is where you want another service to trigger your Lambda function in response to some event. For example, you might want to trigger your Lambda function when a new item is added to a particular S3 bucket, or when events arrive in a Kinesis stream. To do this, the Lambda function must use a "resource-based policy". This policy gives other services the permission to invoke your Lambda function.
There are a few ways to work with other AWS services.
If the service you are accessing is an AWS RDS Database, like SQL Server, or Postgres, you use the same libraries you would use if you hosted the databases on your own computer or data center. You need a connection string with username and password, or another form of authentication of your choice. There is nothing different from your day-to-day usage of the database. You don't need any additional permissions to access the database server. The only caveat to this is if the database is not publicly accessible, then you need to connect the Lambda to the VPC (that process requires extra permissions).
If your Lambda function is using S3, DynamoDB, Kinesis, etc, you use the AWS SDKs to interact with those services. The role your Lambda function is running under needs appropriate permissions to interact with each service. For example, if you want to add an item to an S3 bucket, the role will need permission to write to that bucket. If you want to get items from a DynamoDB table, the role will need permissions to read from that table.
The third scenario is where you want another service to trigger your Lambda function in response to some event. For example, you might want to trigger your Lambda function when a new item is added to a particular S3 bucket, or when events arrive in a Kinesis stream. To do this, the Lambda function must use a "resource-based policy". This policy gives other services the permission to invoke your Lambda function.
Time to Complete
30 minutes
Accessing RDS Database Servers from a Lambda function
The great thing about using familiar services like SQL Server, Postgres, MySQL, is that from a code perspective, you don't need to do anything differently when calling them from a Lambda function. Entity Framework/ADO/NpgSql, etc., work just as well with an AWS hosted database as they do with a local/racked database. You call it the same way, you don't need an AWS SDK library, of course you still need to add the relevant NuGet packages to your project. But otherwise, it's all the same.
Accessing AWS Services from a Lambda function
To access most AWS Services, you need to give the Lambda function permission to interact with that service. You do this by adding a policy to the role that the Lambda function is running under. The policy will need to grant the Lambda function permission to interact with the service. You can create the policy in two ways –
1. as an inline policy where you use it only with that role.
2. As a stand-alone policy that you can attach to any role. In AWS the latter is called a customer-managed policy.
It is always good practice to give the role the least permissions possible. In the following example, where you will read from the DynamoDB table, you need to grant the Lambda role two permissions - dynamodb:GetItem and dynamodb:DescribeTable. And you will limit those permissions to the specific table you are interested in.
First, create a new DynamoDB table called People. The following commands will work with PowerShell, if you are using the Windows command prompt, or Linux shell will need different escaping for the strings.
Run the following -
2. As a stand-alone policy that you can attach to any role. In AWS the latter is called a customer-managed policy.
It is always good practice to give the role the least permissions possible. In the following example, where you will read from the DynamoDB table, you need to grant the Lambda role two permissions - dynamodb:GetItem and dynamodb:DescribeTable. And you will limit those permissions to the specific table you are interested in.
First, create a new DynamoDB table called People. The following commands will work with PowerShell, if you are using the Windows command prompt, or Linux shell will need different escaping for the strings.
Run the following -
aws dynamodb create-table --table-name People --attribute-definitions AttributeName=PersonId,AttributeType=N --key-schema AttributeName=PersonId,KeyType=HASH --provisioned-throughput ReadCapacityUnits=1,WriteCapacityUnits=1
Note the TableArn, you will use it in a moment. It will be near the bottom of the output.
Add a few items to the table:
Add a few items to the table:
aws dynamodb put-item --table-name People --item '{"PersonId":{"N":"1"},"State":{"S":"MA"}, "FirstName": {"S":"Alice"}, "LastName": {"S":"Andrews"}}'
aws dynamodb put-item --table-name People --item '{"PersonId":{"N":"2"},"State":{"S":"MA"}, "FirstName": {"S":"Ben"}, "LastName": {"S":"Bradley"}}'
aws dynamodb put-item --table-name People --item '{"PersonId":{"N":"3"},"State":{"S":"MA"}, "FirstName": {"S":"Claire"}, "LastName": {"S":"Connor"}}'
You now have a table with three items in it.
Next, create a Lambda function using:
Next, create a Lambda function using:
dotnet new lambda.EmptyFunction -n LambdaFunctionDynamoDB
Change to the LambdaFunctionDynamoDB /src/LambdaFunctionDynamoDB directory, and add the AWSSDK.DynamoDBv2 NuGet package:
cd LambdaFunctionDynamoDB /src/LambdaFunctionDynamoDB
dotnet add package AWSSDK.DynamoDBv2
Then open the Function.cs and replace the code with the following:
using Amazon.DynamoDBv2;
using Amazon.DynamoDBv2.DataModel;
using Amazon.Lambda.Core;
[assembly: LambdaSerializer(typeof(Amazon.Lambda.Serialization.SystemTextJson.DefaultLambdaJsonSerializer))]
namespace LambdaFunctionDynamoDB ;
public class Function
{
public async Task<string> FunctionHandler(ILambdaContext lambdaContext)
{
AmazonDynamoDBConfig clientConfig = new AmazonDynamoDBConfig();
AmazonDynamoDBClient client = new AmazonDynamoDBClient(clientConfig);
DynamoDBContext dynamoDbContext = new DynamoDBContext(client);
Person person = await dynamoDbContext.LoadAsync<Person>(1);
return $"{person.FirstName} {person.LastName} lives in {person.State}";
}
}
[DynamoDBTable("People")]
public class Person
{
[DynamoDBHashKey]
public int PersonId {get; set;}
public string State {get; set;}
public string FirstName {get; set;}
public string LastName {get; set;}
}
Deploy the Lambda function to AWS Lambda using:
dotnet lambda deploy-function LambdaFunctionDynamoDBDeploy the Lambda function to AWS Lambda using:
dotnet lambda deploy-function LambdaFunctionDynamoDB
Next you will be asked "Select IAM Role that to provide AWS credentials to your code:", you may be presented with a list of roles you created previously, but at the bottom of the list will be the option "*** Create new IAM Role ***", type in that number beside that option.
You will be asked to "Enter name of the new IAM Role:". Type in "LambdaFunctionDynamoDBRole".
You will then be asked to "Select IAM Policy to attach to the new role and grant permissions" and a list of policies will be presented. Select AWSLambdaBasicExecutionRole, it is number 6 on my list. (I know there is a policy called AWSLambdaDynamoDBExecutionRole, but the goal of this module is to show you how to add the necessary permissions yourself).
Try to invoke the Lambda function using:
dotnet lambda invoke-function LambdaFunctionDynamoDB
You will get a long error message, but very near the top you will see something like -
"errorMessage": "User: arn:aws:sts::YOUR_ACCOUNT_NUMBER:assumed-role/LambdaFunctionDynamoDB Role/LambdaFunctionDynamoDB is not authorized to perform: dynamodb:DescribeTable on resource: arn:aws:dynamodb:us-east-1:YOUR_ACCOUNT_NUMBER:table/People because no identity-based policy allows the dynamodb:DescribeTable action"
Note the message: "...no identity-based policy allows the dynamodb:DescribeTable action".
This is telling you that the role the Lambda function is running as doesn't have the required dynamodb:DescribeTable permission.
To fix this, you need to add a policy granting the role the dynamodb:DescribeTable permission. As mentioned above, you can add an inline policy (only for this role), or a stand-alone policy (available to all roles).
Create a file called DynamoDBAccessPolicy.json in the LambdaFunctionDynamoDB /src/LambdaFunctionDynamoDB folder.
Edit DynamoDBAccessPolicy, but use your account number in the resource:
This is telling you that the role the Lambda function is running as doesn't have the required dynamodb:DescribeTable permission.
To fix this, you need to add a policy granting the role the dynamodb:DescribeTable permission. As mentioned above, you can add an inline policy (only for this role), or a stand-alone policy (available to all roles).
Create a file called DynamoDBAccessPolicy.json in the LambdaFunctionDynamoDB /src/LambdaFunctionDynamoDB folder.
Edit DynamoDBAccessPolicy, but use your account number in the resource:
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": [
"dynamodb:DescribeTable"
],
"Resource": "arn:aws:dynamodb:us-east-1:YOUR_ACCOUNT_NUMBER:table/People"
}
]
}
Run the following:
aws iam put-role-policy --role-name LambdaFunctionDynamoDBRole --policy-name LambdaFunctionDynamoDBAccess --policy-document file://DynamoDBAccessPolicy.json
It can take a while for the policy to go into effect. You can wait a few minutes, or redeploy the Lambda function. If you want to redeploy the function run:
dotnet lambda deploy-function LambdaFunctionDynamoDB
After the Lambda function is deployed, you can invoke it again using:
dotnet lambda invoke-function LambdaFunctionDynamoDB
Another error!
This time the message is:
This time the message is:
"errorMessage": "User: arn:aws:sts::YOUR_ACCOUNT_NUMBER:assumed-role/LambdaFunctionDynamoDB Role/LambdaFunctionDynamoDB is not authorized to perform: dynamodb:GetItem on resource: arn:aws:dynamodb:us-east-1:YOUR_ACCOUNT_NUMBER:table/People because no identity-based policy allows the dynamodb:GetItem action",
You need to add "dynamodb:GetItem" to the array of permissions in the policy.
Update the DynamoDBAccessPolicy.json file with the following -
Update the DynamoDBAccessPolicy.json file with the following -
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": [
"dynamodb:DescribeTable",
"dynamodb:GetItem"
],
"Resource": "arn:aws:dynamodb:us-east-1:YOUR_ACCOUNT_NUMBER:table/People"
}
]
}
Redeploy the Lambda function:
dotnet lambda deploy-function LambdaFunctionDynamoDB
Invoke again:
dotnet lambda invoke-function LambdaFunctionDynamoDB
Success!
Amazon Lambda Tools for .NET Core applications (5.4.2)
Project Home: https://github.com/aws/aws-extensions-for-dotnet-cli, https://github.com/aws/aws-lambda-dotnet
Payload:
"Alice Andrews lives in MA"
Lambda error messages will be very helpful when trying to give the least permissions possible to a role, but as you saw, you might have to repeat this process a few times.
Another option is to hover over the SDK method you are using, the metadata may contain useful information about permissions. Not all method metadata will contain permission information.
Another option is to hover over the SDK method you are using, the metadata may contain useful information about permissions. Not all method metadata will contain permission information.
You should also consult the AWS service documentation for information on required permissions.
Now you know how to find out what permissions your function needs and how to grant the correct permissions to the role your Lambda functions are running as.
Now you know how to find out what permissions your function needs and how to grant the correct permissions to the role your Lambda functions are running as.
Allowing Other Services to Invoke Lambda Functions
You will have many scenarios where you need another service to invoke a Lambda function for you. Some common scenarios include the arrival of a message in a queue, arrival of an event in a Kinesis stream, a change to an object/bucket in S3, a request arriving to an API Gateway. In each of these cases, the Lambda function will be invoked by another AWS service.
In the previous section, you learned about giving permissions to the Lambda function to perform actions on other services. In this section, you will see how to give other services permissions to invoke your Lambda function.
If you are using the serverless.* templates, you are probably already giving an API Gateway the required permission to invoke your Lambda function. If you have deployed such a function, go to the Configuration tab, then select Permissions on the left, and scroll to the Resource-based policy section. You will see policies allowing the API Gateway to invoke your Lambda function. This policy was added by the dotnet lambda deploy-serverless command, and the serverless.template in your project.
In the image below, you can see two policy statements allowing an API Gateway to invoke the Lambda function.
In the previous section, you learned about giving permissions to the Lambda function to perform actions on other services. In this section, you will see how to give other services permissions to invoke your Lambda function.
If you are using the serverless.* templates, you are probably already giving an API Gateway the required permission to invoke your Lambda function. If you have deployed such a function, go to the Configuration tab, then select Permissions on the left, and scroll to the Resource-based policy section. You will see policies allowing the API Gateway to invoke your Lambda function. This policy was added by the dotnet lambda deploy-serverless command, and the serverless.template in your project.
In the image below, you can see two policy statements allowing an API Gateway to invoke the Lambda function.
But the example you will work on allows an S3 bucket to invoke your Lambda function whenever you create or delete a file in that bucket.
Create the S3 Bucket
First step is to create an S3 bucket.
If you want your bucket in us-east-1, you can use the following command -
If you want your bucket in us-east-1, you can use the following command -
aws s3api create-bucket --bucket my-unique-bucket-name-lambda-course
If you want your bucket in a different region, you can use the following command -
aws s3api create-bucket --bucket my-unique-bucket-name-lambda-course --create-bucket-configuration LocationConstraint=REGION
For a list of valid regions for the LocationConstraint, see the following link -
https://awscli.amazonaws.com/v2/documentation/api/latest/reference/s3api/create-bucket.html.
Note, the bucket name must be unique. See here for more
Create the Lambda Function
There is a Lambda function template that will help you handle S3 events. It has the required SDK NuGet packages already added. You still need to add the required role permission and create the resource-based policy to allow S3 to invoke the function.
From the command line run:
From the command line run:
dotnet new lambda.S3 -n S3EventHandler
Change to the S3EventHandler/src/S3EventHandler directory:
cd S3EventHandler/src/S3EventHandler
Open the Function.cs file, replace the FunctionHandler method with the following:
public async Task FunctionHandler(S3Event evnt, ILambdaContext context)
{
context.Logger.LogInformation($"A S3 event has been received, it contains {evnt.Records.Count} records.");
foreach (var s3Event in evnt.Records)
{
context.Logger.LogInformation($"Action: {s3Event.EventName}, Bucket: {s3Event.S3.Bucket.Name}, Key: {s3Event.S3.Object.Key}");
if (!s3Event.EventName.Contains("Delete"))
{
try
{
var response = await this.S3Client.GetObjectMetadataAsync(s3Event.S3.Bucket.Name, s3Event.S3.Object.Key);
context.Logger.LogInformation( $"The file type is {response.Headers.ContentType}");
}
catch (Exception e)
{
context.Logger.LogError(e.Message);
context.Logger.LogError($"An exception occurred while retrieving {s3Event.S3.Bucket.Name}/{s3Event.S3.Object.Key}. Exception - ({e.Message})");
}
}
else
{
context.Logger.LogInformation($"You deleted {s3Event.S3.Bucket.Name}/{s3Event.S3.Object.Key}");
}
}
}
When the Lambda function receives an S3 event, the Lambda function will log the event details to CloudWatch. If the S3 event is in response to an object being created, the function will use the AWS SDK to make a call to S3 to get the file type, and then log the details.
If the S3 event is in response to an object being deleted, the function will log the bucket/key names to CloudWatch.
If the S3 event is in response to an object being deleted, the function will log the bucket/key names to CloudWatch.
Deploy the Lambda Function
From the command line run:
dotnet lambda deploy-function S3EventHandler
Next you will be asked "Select IAM Role that to provide AWS credentials to your code:", you may be presented with a list of roles you created previously, but at the bottom of the list will be the option "*** Create new IAM Role ***", type in that number beside that option.
You will be asked to "Enter name of the new IAM Role:". Type in "S3EventHandlerRole".
You will then be asked to "Select IAM Policy to attach to the new role and grant permissions" and a list of policies will be presented. Select AWSLambdaBasicExecutionRole, it is number 6 on my list. You will need to add a policy to grant access to the S3 bucket for the GetObjectMetadataAsync(..) call to work.
Grant the Lambda Function Permissions to Get the Object Metadata
In the previous example, you created an inline policy that applied to only the role you were using. This time, you will create a policy that any role can use.
You will see how to do this in a couple of ways.
You will see how to do this in a couple of ways.
Command line
The policy will look like this, but with your bucket name in the resource. Note the /* at the end, this means s3:GetObject applies to all objects in the bucket:
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": [
"s3:GetObject"
],
"Resource": "arn:aws:s3:::my-unique-bucket-name-lambda-course/*"
}
]
}
Run the following:
aws iam create-policy --policy-name S3AccessPolicyForCourseBucket --policy-document file://S3AccessPolicyForCourseBucket.json
Take note of the ARN of the policy.
Then attach the policy to the role you created earlier. Run the following:
Then attach the policy to the role you created earlier. Run the following:
aws iam attach-role-policy --role-name S3EventHandlerRole --policy-arn arn:aws:iam::694977046108:policy/S3AccessPolicyForCourseBucket
AWS Console
Click on the Configuration tab, then Permissions on the left, and click the name of the role.
This opens a new page with details of the role.
Click Add permissions, and Attach policies.
Click Add permissions, and Attach policies.
Click Create policy
In the service section, type s3 into the highlighted text box, and select S3.
In the Actions section, type in getobject and select GetObject from the list.
In the resources section, choose Specific, and click Add ARN.
Type in the bucket name, and select Any for the object name.
Type in a name for the policy, and click Create policy
Go back to the tab where you clicked Create policy. Follow these steps -
1. Reload the list of policies
2. Type in S3AccessPolicyForCourseBucket in the filter
3. Tick the box beside the policy
4. Click Attach policies
At this point, you have an S3 bucket, the Lambda function, and required permissions to get the object metadata from the S3 bucket.
Now it's time to wire the S3 bucket to the Lambda function, so creation and deletion events trigger the Lambda function.
Trigger the Lambda Function from the S3 Bucket
As you have seen, it's good to be able to use the AWS command-line tools and the UI Console. For the next step, you are going to use the UI Console, because it is easier and clearer for this step.
Open the list of buckets in S3 https://s3.console.aws.amazon.com/s3/buckets.
Click on the one you created.
Open the list of buckets in S3 https://s3.console.aws.amazon.com/s3/buckets.
Click on the one you created.
Open the Properties tab.
Scroll down to the Event notifications section.
Click Create event notification.
Put in a name for the event notification.
Select the first two tick boxes on the left - All object create events, and All object removal events.
Scroll to the Destination section at the bottom.
Choose Lambda function as the destination.
In the dropdown list, type in the name of the Lambda function you created earlier.
Click Save changes.
When done, you will see this new event notification listed in the Event notifications section.
In the AWS Console, go to the Lambda function you created earlier.
Note that S3 is now listed as a trigger for the Lambda function.
Click on the Configuration tab, and then Permissions on the left.
Scroll down to the Resource-based policy section
You will see a policy statement allowing S3 to invoke the Lambda function.
You will see a policy statement allowing S3 to invoke the Lambda function.
Click on the Statement ID to review the policy statement.
Testing it out
This Lambda function doesn't return anything to the caller, S3 in this case.
Instead, the Lambda function logs to CloudWatch, so it's there you have to go to see your function working.
Create a text file on your computer to upload to S3.
From the command line, run:
Instead, the Lambda function logs to CloudWatch, so it's there you have to go to see your function working.
Create a text file on your computer to upload to S3.
From the command line, run:
aws s3api put-object --bucket my-unique-bucket-name-lambda-course --key Hello.txt --body Hello.txt --content-type "text/plain"
aws s3api delete-object --bucket my-unique-bucket-name-lambda-course --key Hello.txtNow go to your Lambda function in the AWS Console and check the logs.
Click the Monitor tab, and then View Logs in CloudWatch.
You will be presented with a list of log streams, choose the most recent one.
You should see log entries that show the S3 events and the text you logged to CloudWatch.
Another way to check the logs is to use the AWS extension for Visual Studio, Visual Studio Code, and Rider.
The process is similar for all three, open the AWS extension, click on CloudWatch logs, and find the log stream/group for /aws/lambda/S3EventHandler. Then open the most recent stream.
The process is similar for all three, open the AWS extension, click on CloudWatch logs, and find the log stream/group for /aws/lambda/S3EventHandler. Then open the most recent stream.
Another way to check the logs is to use the AWS extension for Visual Studio, Visual Studio Code, and Rider.
The process is similar for all three, open the AWS extension, click on CloudWatch logs, and find the log stream/group for /aws/lambda/S3EventHandler. Then open the most recent stream.
The process is similar for all three, open the AWS extension, click on CloudWatch logs, and find the log stream/group for /aws/lambda/S3EventHandler. Then open the most recent stream.
Conclusion
In this relatively short module, you have covered a lot of ground. Some would say that understanding roles and policies is one of the most important things to learn on AWS. Hopefully, this has given you a good grounding in the topic in relation to Lambda functions.
Here is the key takeaway, if you want your Lambda function to interact with other AWS services, you need to give your function permissions to act on that other service.
If you want other services to invoke your function, you need to use resource-based policies to give those services access to your function.
Here is the key takeaway, if you want your Lambda function to interact with other AWS services, you need to give your function permissions to act on that other service.
If you want other services to invoke your function, you need to use resource-based policies to give those services access to your function.
Knowledge Check
You’ve now completed Module 4, Working With Other AWS Services. The following test will allow you to check what you’ve learned so far.
1. When you want another service to invoke a Lambda function what do you need to do? (select one)
a. Create an ACL document giving the role the other service runs as permission to invoke the Lambda function5
b. Create resource-based policy document giving the calling services permission to invoke the Lambda function
c. Nothing, the Lambda trusts all other AWS services
d. Add the correct permissions to the role the Lambda function runs as1
b. Create resource-based policy document giving the calling services permission to invoke the Lambda function
c. Nothing, the Lambda trusts all other AWS services
d. Add the correct permissions to the role the Lambda function runs as1
2. What do you need to add to a role to give it permissions to access AWS services? (select one)
a. Nothing, all roles can access other AWS services.
b. A resource-based policy
c. A policy with the necessary permissions
d. An access control list document
b. A resource-based policy
c. A policy with the necessary permissions
d. An access control list document
3. What are two ways to create a customer-managed policy for use with the role a Lambda function runs as? (select two)
a. Via the command line
b. Include in the source code of the function
c. Via that AWS Console
d. Add it to the payload when executing the function
Answers: 1-b, 2-c, 3-ac
Conclusion
In this relatively short module, you have covered a lot of ground. Some would say that understanding roles and policies is one of the most important things to learn on AWS. Hopefully, this has given you a good grounding in the topic in relation to Lambda functions.
Here is the key takeaway, if you want your Lambda function to interact with other AWS services, you need to give your function permissions to act on that other service.
If you want other services to invoke your function, you need to use resource-based policies to give those services access to your function.
Here is the key takeaway, if you want your Lambda function to interact with other AWS services, you need to give your function permissions to act on that other service.
If you want other services to invoke your function, you need to use resource-based policies to give those services access to your function.