Build a Serverless Lie Detector that uses AI for Facial Recognition

GitHub Repository: https://github.com/shreythecray/lie-detector Follow along with the video tutorial: https://youtu.be/0RJ9m-T7sgU We will build a serverless application that uses Artificial Intelligence to conduct facial recognition and uses Courier to send alerts based on the results. We just launched our first hackathon and are giving away over $1K in prizes! Join us in building a cool project and winning any of the following prizes. 🏆 Top submission for use for notifications and demonstration of proper app-to-user notifications with the Courier API will receive $1000 via Zelle or PayPal. Courier Hacks 1st Place: 2nd Place submission for use for notifications and demonstration of proper app-to-user notifications with the Courier API will receive the Apple AirPods Pro. Courier Hacks 2nd Place: Public favorite winner will receive a Keychron Keyboard. Public Favorite: Runners-up submissions will receive a Hydro Flask. Runners-Up: Additionally, everyone who submits a project successfully integrating the Courier API will receive a $20 Amazon gift card! Submissions close on September 28th. Register now to submit this project for a chance to win some cool prizes. Register for the Hackathon: https://courier-hacks.devpost.com/ Not sure where to start? In this tutorial, we will build a serverless lie detector that uses Artificial Intelligence for Facial Recognition. What’s going on? Let’s get started. We are secret agents and headquarters is telling us that one of our spies has betrayed us and has been leaking sensitive, top-secret information to our enemies. Our goal is to build a lie detector that will alert our spy network when we identify the mole. We will use Azure's Cognitive Services to perform facial recognition on everyone on our team. When the Face API recognizes that one of our spies is being deceitful, we will use Courier to broadcast the identity of the mole to our spy network. Some spies are in an area where they can only guarantee secure messaging through emails, and others prefer quick and secure SMS, so we need to ensure our app can accommodate all spy preferences. The first three Secret Agents to successfully complete this tutorial and task will receive a top-secret gift from HQ via Courier. Note: In Part 1, we will create our serverless application using Azure Functions. In Part 2, we will first integrate the Gmail and Twilio APIs, which Courier will use to send emails and text messages. In Part 3, we will demonstrate how to send single messages and set up routing to send multi-channel notifications from the Azure function. And finally, in Part 4, we will explore Azure Cognitive Services and integrate the Face API to analyze emotions and use the Courier API to send alerts when specific deceiving emotions are detected. Instructions Part 1: Create a serverless application using Azure Functions We first need to set up our local development environment to enable using Azure and testing our code. Install the following VS Code Extensions: Azure Tools to create Azure Functions, REST Client to test our Azure Function calls (this is an alternative to Postman or Insomnia). Once the two extensions have been installed successfully, check the left menu for Azure’s A symbol. Completely close and reopen VS Code if the symbol does not automatically appear. To build an HTTP Trigger Function: Click on the Azure symbol to open Azure locally. Here we are prompted to sign into our Azure account. After signing in, we need to open an Azure Subscription or create a new one. Note that students can get free credits for Azure. If the subscription is not showing up locally, follow instructions from the , which tells us to sign out of Azure on VS Code and sign back in. documentation on how to set up our account Once the subscription appears, click on “Workspace” and create a new project. Determine where we need to save the files within this project and how we want to name it locally. Once the location has been selected, we are prompted to make a few decisions about the type of function we want to create. Select JavaScript for the language Select HTTP Trigger for template Rename the function to LieDetector (or any unique name) Select Function for the Authorization Level. Open the project to start coding. You can edit settings later within the function.json file. Let’s open the index.js file and take a moment to understand the boilerplate function. module.exports = async function (context, req) { context.log('JavaScript HTTP trigger function processed a request.'); const name = (req.query.name || (req.body && req.body.name)); const responseMessage = name ? "Hello, " + name + ". This HTTP triggered function executed successfully." : "This HTTP triggered function executed successfully. Pass a name in the query string or in the request body for a personalized response."; context.res = { // status: 200, /* Defaults to 200 */ body: responseMessage }; } Line #4 demonstrates how we can get data from the function query or request body. Line #5 contains a string assigned to a variable named , which uses the data from line #4. On lines #9-12, this variable is then passed into the response object, . responseMessage context.res Part 2: Authorize Courier to send messages using Gmail and Twilio APIs Courier sends messages by consolidating multiple API calls into one. In this second part, we will need to authorize our API to send messages via the Gmail and Twilio APIs. and create a new secret workspace. Log in to your Courier account For onboarding, select the email channel and let Courier build with Node.js. Start with the Gmail API since it only takes seconds to set up. All we need to do to authorize is log in via Gmail. Now the API is ready to send messages. Copy the starter code, a basic API call using cURL, and paste it into a new terminal. It already has saved your API key, knows which email address you want to send to, and has a built-in message. Once you see Agent Pigeon dancing, we are ready to use Courier to communicate with our spies. Before we build out our application, we need to set up the Twilio provider to enable text messages. Head over to “Channels" in the left menu and search for Twilio. You will need an Account SID, Auth Token, and a Messaging Service SID to authorize Twilio. Open , login and open the console, and find the first two tokens on that page. Save the Account SID and Auth Token in Courier. twilio.com Lastly, you need to locate the Messaging Service SID, which you create in the Messaging tab on the left menu. Checkout Twilio’s docs on , linked in the description. how to create a Messaging Service SID Once we have all three pieces of information, install the provider. Your Courier account is officially authorized to send any email or SMS within one API call. Part 3: Send single and multi-channel notifications from the Azure function We can use Courier to send single messages or set up routing to send multi-channel notifications from within the Azure function. In this third Part, we will start sending messages and will refer to the Courier Node.js quick start, which outlines how to get started with the SDK, and can be found within the Docs page on . courier.com If you would rather use the Courier API, check out the for instructions or for documentation. Secret Message tutorial Courier’s API Reference The walks us through getting access to the API key. SDK documentation Find the API key on . https://app.courier.com/settings/api-keys Save the API key in the local.settings.json file and bring it into the index.js file as . This application can now be authorized to use our Courier account. const apiKey = process.env["API_KEY"] { "IsEncrypted": false, "Values": { "AzureWebJobsStorage": "", "FUNCTIONS_WORKER_RUNTIME": "node", "API_KEY": "replace-with-your-key" } } Install the Courier SDK by running the following command in the terminal: npm install @trycourier/courier Use the require function to import the SDK into the index.js file. const { CourierClient } = require("@trycourier/courier"); The last step is to walk through the API call within the API docs and integrate it into our codebase. import { CourierClient } from "@trycourier/courier"; // alternatively: // const { CourierClient } = require("@trycourier/courier"); const courier = CourierClient({ authorizationToken: "YOUR_AUTH_TOKEN_HERE" }); const { requestId } = await courier.send({ message: { to: { email: "email@example.com", }, content: { title: "Welcome!", body: "Thanks for signing up, {{name}}", }, data: { name: "Peter Parker", }, routing: { method: "single", channels: ["email"], }, }, }); Copy line #5 and paste it above and outside the async function (Azure function). Everything within lines #7-24 is related to the actual message sending, and needs to be placed inside the async function. The code on lines #8-23 defines the message object, which provides data to Courier about the messages: the object about the user receiving the notification, the object about what the message contains, the object about any variables that impact the object or conditions for the outgoing notifications, and the object about the types of notifications being sent. to content data content routing Update the email that this message is being sent to. To protect the identities of our spies, we will use fake contact information here. Update the message here. For example, we can change the or email subject to and the to In this case, we can either hardcode the name or get it from the HTTP trigger function body. title Mule Identified body Beware! The mule is {{name}}. Update the and log it to the console. This new will indicate to us that this HTTP triggered function runs successfully by outputting the response from the Courier API call. responseMessage responseMessage requestId To run this function locally, first run the command in the terminal, enabling the trigger for this function (and all functions within this project, if there were others). This command will also return to us the corresponding local endpoint that we can use to trigger this function. func start const { requestId } = await courier.send({ message: { to: { email: "courier.demos+liedetector@gmail.com", }, content: { title: "Mule Identified!", body: "Beware! The mule's name is {{name}}.", }, data: { name: name, }, routing: { method: "single", channels: ["email"], }, }, }); We can use Postman or Insomnia to test this function. Here we will use the REST Client VS Code extension, which we installed earlier Create a request.http file. To create a new test call, type at the top of the request.http file. ### Below, define the type of request, in this case , and paste the endpoint next to it. POST The body of this function call still needs to be defined under the endpoint. Create an object that contains a parameter and define it as name Secret Agent Pigeon. ### POST http://localhost:7071/api/LieDetector { "name": "Secret Agent Pigeon" } Part 4: Analyze emotions with the Face API and send alerts with Courier Azure Cognitive Services enables us to add cognitive capabilities to apps through APIs and AI services. In this final part, we will explore Azure Cognitive Services, integrate the Face API to analyze emotions, and use the Courier API to send alerts when specific deceiving emotions are detected. Check out all of the services provided. To access the Face API, navigate to the . Azure portal “Create a Resource” and locate “AI + Machine Learning” in the list of categories on the left. Select “Face” from the list of services that appear and update the settings based on our preferences and account. Hit “Review + Create” and then “Create” to begin the deployment process (this takes a few minutes to complete). Once deployment is complete, head over to the resource, navigate to “Keys and Endpoints” on the left menu under “Resource Management”, copy one of the keys and the endpoint, and save them within our project in the local.settings.json file. { "IsEncrypted": false, "Values": { "AzureWebJobsStorage": "", "FUNCTIONS_WORKER_RUNTIME": "node", "API_KEY": "replace-with-your-key", "FACE_API_KEY": "replace-with-your-azure-key", "FACE_ENDPOINT": "replace-with-your-azure-endpoint" } } These values are treated as secret keys and this file is included in the .gitignore. Just as we used the Courier SDK, we will use the Face service with Azure’s SDK, which can be found on the . We will need to use both commands on this page. Azure Cognitive Services npm page Run in the terminal to install the Face service SDK. npm install @azure/cognitiveservices-face Run in the terminal to install the REST Azure Client. npm install @azure/ms-rest-azure-js Copy the sample Face API call from the sample code and place it into our codebase. Move the import statements to the top, above the Azure Function. const { FaceClient, FaceModels } = require("@azure/cognitiveservices-face"); const { CognitiveServicesCredentials } = require("@azure/ms-rest-azure-js"); async function main() { const faceKey = process.env["faceKey"] || " "; const faceEndPoint = process.env["faceEndPoint"] || " "; const cognitiveServiceCredentials = new CognitiveServicesCredentials(faceKey); const client = new FaceClient(cognitiveServiceCredentials, faceEndPoint); const url = "https://pbs.twimg.com/profile_images/3354326900/3a5168f2b45c07d0965098be1a4e3007.jpeg"; const options = { returnFaceLandmarks: true }; client.face .detectWithUrl(url, options) .then(result => { console.log("The result is: "); console.log(result); }) .catch(err => { console.log("An error occurred:"); console.error(err); }); } main(); Now, we can update the variables within the code snippet. Update the name of the function on line #43 and the associated function call on line #65 to . analyze_face() Fix the key names on lines #44 and #45 to match the names we created in the local.settings.json file. Line #49 contains the image this API will analyze. Find a link to an image of our own - to protect the identities of our spies, we will use IU’s image. Change the object between lines #50 to #52 to and add an array with as an element. options returnFaceAttributes emotion async function analyze_face() { const faceKey = process.env["FACE_API_KEY"]; const faceEndPoint = process.env["FACE_ENDPOINT"]; const cognitiveServiceCredentials = new CognitiveServicesCredentials(faceKey); const client = new FaceClient(cognitiveServiceCredentials, faceEndPoint); const url = "https://www.allkpop.com/upload/2021/12/content/231225/web_data/allkpop_1640280755_untitled-1.jpg"; const options = { returnFaceAttributes: ["emotions"] }; client.face .detectWithUrl(url, options) .then(result => { console.log("The result is: "); console.log(result); }) .catch(err => { console.log("An error occurred:"); console.error(err); }); } analyze_face(); Finally, we need to be able to manipulate the response from this API call. Save the response in a variable called . result Convert the response into a JSON string using the method. stringify const result = await client.face.detectWithUrl(url, options); const resultJSON = JSON.stringify(result, null, 2); We run into an error when we use the REST Client to test our function. The result is not displayed, which means that for some reason, is not returning the correct response. We can check the Face API reference to determine the cause of the error. We can first attempt to resolve the issue by removing a specific from the result object. analyze_face() emotion const result = await client.face.detectWithUrl(url, options); const anger = result[0].faceAttributes.emotion.anger; const angerJSON = JSON.stringify(anger, null, 2); The actual error stems from a typo on line #51 where the object returned is not plural and should be called . When we test the code, we see that the anger emotion has a value of 0, which matches the selected image. emotion const options = { returnFaceAttributes: ["emotion"] }; Finally, instead of returning the value of a single emotion, update the function to return the entire object. This will enable us to compare multiple emotions' values and determine whether the face being analyzed is deceitful. analyze_face() emotion const result = await client.face.detectWithUrl(url, options); return result[0].faceAttributes.emotion; Following instructions from Headquarters, we know that our questions should only invoke specific reactions. If a face shows any hint of anger, neutral, or contempt emotions, we will have to assume that the person being questioned is the mule. Extract these emotions. const emotions = await analyze_face(); const anger = emotions.anger; const angerJSON = JSON.stringify(anger, null, 2); const neutral = emotions.neutral; const neutralJSON = JSON.stringify(neutral, null, 2); const contempt = emotions.contempt; const contemptJSON = JSON.stringify(contempt, null, 2); With a simple condition, compare them to 0. if((angerJSON > 0)||(neutralJSON > 0)||(contemptJSON > 0)) { deceptive = true; } Send out alerts to our entire spy network if and when these values are larger than 0. const { CourierClient } = require("@trycourier/courier"); const { FaceClient, FaceModels } = require("@azure/cognitiveservices-face"); const { CognitiveServicesCredentials } = require("@azure/ms-rest-azure-js"); const apikey = process.env["API_KEY"]; const courier = CourierClient({ authorizationToken: apikey }); module.exports = async function (context, req) { context.log('JavaScript HTTP trigger function processed a request.'); const name = (req.query.name || (req.body && req.body.name)); const emotions = await analyze_face(); const anger = emotions.anger; const angerJSON = JSON.stringify(anger, null, 2); const neutral = emotions.neutral; const neutralJSON = JSON.stringify(neutral, null, 2); const contempt = emotions.contempt; const contemptJSON = JSON.stringify(contempt, null, 2); let deceptive = false; if((angerJSON > 0)||(neutralJSON > 0)||(contemptJSON > 0)) { deceptive = true; } if(deceptive) { const { requestId } = await courier.send({ message: { to: { email: "courier.demos+liedetector@gmail.com", }, content: { title: "Mule Identified!", body: "Beware! The mule's name is {{name}}.", }, data: { name: name, }, routing: { method: "single", channels: ["email"], }, }, }); } const responseMessage = "The HTTP trigger function ran successfully."; context.res = { // status: 200, /* Defaults to 200 */ body: { responseMessage, "anger": angerJSON, "neutral": neutralJSON, "contempt": contemptJSON } }; } async function analyze_face() { const faceKey = process.env["FACE_API_KEY"]; const faceEndPoint = process.env["FACE_ENDPOINT"]; const cognitiveServiceCredentials = new CognitiveServicesCredentials(faceKey); const client = new FaceClient(cognitiveServiceCredentials, faceEndPoint); const url = "https://www.allkpop.com/upload/2021/12/content/231225/web_data/allkpop_1640280755_untitled-1.jpg"; const options = { returnFaceAttributes: ["emotion"] }; const result = await client.face.detectWithUrl(url, options) return result[0].faceAttributes.emotion; } Conclusion Our lie detector is ready and will alert our spies anytime a captive tries to mess with us. Try building a lie detector of your own and alerting courier.demos+liedetector@gmail.com, and we will send a gift to the first three Secret Agents to complete this task! Head to to get started. Don’t forget to submit your project to our for a chance to win over $1000 in cash and prizes! courier.com/hack-now Hackathon Quick Links 🔗 GitHub Repository: https://github.com/shreythecray/lie-detector https://youtu.be/0RJ9m-T7sgU 🔗 Courier: app.courier.com 🔗 Register for the Hackathon: https://courier-hacks.devpost.com/ 🔗 Courier's Get Started with Node.js: https://www.courier.com/docs/guides/getting-started/nodejs/ 🔗 Courier Send API Docs: https://www.courier.com/docs/reference/send/message/ 🔗 Twilio Messaging Service SID Docs: https://support.twilio.com/hc/en-us/articles/223181308-Getting-started-with-Messaging-Services 🔗 Courier API Reference: https://www.courier.com/docs/reference/ 🔗 Azure for Students: https://azure.microsoft.com/en-us/free/students/ 🔗 Troubleshooting Azure Account Setup: https://github.com/microsoft/vscode-azure-account/wiki/Troubleshooting#setup-your-azure-account 🔗 Azure Cognitive Services: https://azure.microsoft.com/en-us/services/cognitive-services/#overview 🔗 Azure Portal: https://portal.azure.com/ 🔗 Azure Cognitive Services SDK: https://www.npmjs.com/package/@azure/cognitiveservices-face