Introduction To Web Applications: Part 2

17 min readSep 29, 2021

RESTful APIs, Backend Design and Architecture, Frontend, and Recommendation Engine (and code deep dive)

This is the second and final part of my introduction to web applications. The first part, which covered an introduction, machine learning explanations, high-level architecture, Python, and databases can be found here: https://negoiddfelix.medium.com/introduction-to-web-applications-part-1-3b8c95550bd6
Naturally, I have also a public repo, where you can see the code for everything we talked about in this series: https://github.com/felixnego/noesis-books

RESTful APIs

REST (Representational State Transfer) is not a technology in itself like almost everything else analyzed in the present paper, but a software architectural pattern. It comprises a set of conventions for how an API should function.
First and foremost is that the retrieval of data happens by accessing a specific URL that the API exposes. This process represents the requesting of an endpoint. The request is made through HTTP and the endpoint is the URL previously mentioned. The latter has to have a certain structure, and this represents another set of REST conventions:

In the example above, the part underlined with red, “https://localhost:5001”, represents the root endpoint and the section underlined in blue is called a path. The path determines what specific resource the client is trying to access and it is tied to a specific method/function in the backend. In order to understand what paths are exposed and what can be retrieved from them, the API documentation should be consulted.

Paths can contain variables and/or query parameters to help access certain resources. In the case of our example, although one cannot tell simply by looking at it, “122” is actually a variable. They are specially marked in the documentation, usually by being preceded by a colon (“/api/reports/topcategories/:id”) or by being surrounded with curly brackets. Query parameters (righthand side example) come in a sequence that starts with a question mark, are presented in key- value pairs (parameter=value) and, if necessary, are separated with an ampersand (“&”). Just like variables, they are parsed and passed into the logic the API is executing when the endpoint is requested.

The combination of path structure and HTTP methods used to reach them is a large part of the RESTful paradigm. The table below describes the most common operations, CRUD, along with the RESTful convention for each of them:

As can be easily seen, patterns emerge both in regards to the method and to the endpoint. Thus, GET is used to read/retrieve data, POST is used to create new data or update existing entries and DELETE is used exactly as its name implies. When one tries to reach a list of entities or create a new one, thought with different methods, the endpoint needed is conventionally just a slash followed by the plural of the entity name. In order to reach a specific entry another slash and its ID are appended.

Let us turn slight to how the two most used methods, GET and POST, handle data transfer.

In both the GET and the POST methods some data is sent over from the client to the server, however the approach is different. As has been exposed previously, a GET request can have variables filled in that the server will parse and use or query parameters right in the URL. This is not a recommended approach for any data that is either sensitive or large. This is why, when new entities are being created, for example, a POST request is used. With such a method, data is sent in the body of the request, not the URL, as the snippet above illustrates.

ASP.NET Core Backend API

No modern, large, scalable application is built with just the standard library of a programming language. This is where frameworks appear to ease the works. The distinction between a library and a framework may be subtle at first, however it is an important one. In a short and pseudo- metaphorical way, a library is code that we import and use whenever we need it. In an inversion of control manner, a framework is something that uses our code, rather than the other way around. It is, perhaps, more accurate to state that our code adapts the framework to our specific use case. A pieces used from a library are wrapped in custom while, while custom code is wrapped by a framework.

ASP.NET Core is a web development framework that offers the advantages of being open source, cross platform and optimized for performance. The .NET platform is highly popular and enjoys a great community, while the C# programming language ought to feel very simple to get accustomed to for developers coming from Java. It supports the creation of a couple of different types of web applications, such as MVC (Model View Controller), though, for the case of Noesis, a web API option has been selected.

Being a rather “opinionated” framework, will scaffold an impressive directory structure and some lines of code that can be used to get started. With regards to the organization of this particular backend API (but also its frontend client, to a certain extent), the layered architecture has been chosen. It provides a much needed separation of concerns, with some degree of decoupling, and proves to be a good complement to the .NET’s framework dependency injection. In abstract, a layered architecture is depicted in the diagram below:

The user interacts with the presentation layer, which represents the UI. Any action they perform triggers the execution of business logic, which is the responsibility of the service. The service will call the repository layer to retrieve or manipulate data as needed. The UI does not perform business logic and the service does not access the database(s) directly.

In the case of the present application, the UI will be represented by the Angular client. In our backend API, we define components for the other two layers. Like many modern applications, Noesis is data driven, which means that the data modeling is a good place to start. Our models are defined as classes with just properties and no methods. Below is an example illustrating the Author model and how the table will be created in the database:

Relationships between entities are also defined at code level, as can be seen from line 22, which shows the implementation of a many-to-many relationship. The peculiar line 11 depicts an annotation can be used to assure data consistency and integrity. However, developers can write their own custom validation as well, using specialized packages, such as Fluent:

To handle the relationship between the MySQL database and the C# models, Noesis uses Microsoft’s Entity Framework (EF) Core. It is a very powerful ORM for the .NET framework. The EF Core provides many tools for handling data, the most important of which is, perhaps, the context. The context is a special class that inherits from DbContext and does all the transactions. It is safe to say that, in the present case, it represents the repository layer of the application. Also, this is where constraints or cascading behavior can be defined for the database’s tables, as the example below depicts:

In order to make the payload transfer between the backend and the frontend more efficient, a common pattern is used involving DTOs (Data Transfer Objects). Their main two uses cases are, firstly, that the backend does not have to send entire objects if only certain fields are sufficient and, secondly, that certain fields need to be manipulated before being sent as part of a response, without wanting to alter the model itself. As mentioned, since the patter is very common, after defining DTO classes, an external package could be used, such as AutoMapper, which helps convert from base objects to DTOs and vice-versa.

A service layer has been defined too, with four classes handling different entities. They use and instance of the context class to interact with the data and perform the business logic of CRUD transactions or creating reports. The service classes do not, however, respond with payloads to the frontend directly. That is the responsibility of the controllers. Illustrated below, a controller is where the endpoints are defined as well as what HTTP methods they accept. They do not, in the case of Noesis, perform any business logic, although for smaller applications that is also feasible to a certain extent. They dispatch that to the different services.

The decorator on line 18 helps define the “/api/books”40 common root for all paths handled inside this class. Line 41 indicates that the function below it is to be called when an HTTP GET request is made. Since no other parameter is passed to “HttpGet”, this will handle the aforementioned root path.

Other decorators can be added to ensure the endpoint is interacted with in the desired manner. For example, “[Authorize(Roles = UserRole.Admin)]” is used to restrict access to administrators only.

The framework is able to bring all of these together with code provided in the Startup.cs file. All application-level configurations are specified here, including, for example, the JWT (JSON Web Token) authentication. While sending login information through HTTPS is technically safe, other methods of authentication and authorization are sometimes preferred. With JWT, the way chosen for Noesis, after a client has made an initial request that was validated by the server, the latter responds with a digitally signed token, that the client keeps, so that the server does not have to keep the data. Every subsequent request from the client is accompanied by the token. The server has to verify that the token has not been tampered with and, if the operation is successful, it then responds with the requested data.

Besides the migrations directory, the API also contains a Helpers folder to store additional functions that may be needed by the workflow. The most pertinent example for such a case is a hashing functions used to store encrypted passwords in the database, and not plain strings, as that would mean a great security risk.

Angular Frontend

Angular is a JavaScript-based frontend framework used in the development of SPAs that works with the concept of re-usable components, like its main alternative, React. The introduction to this paper has briefly explained what SPAs are and how client-side rendering differs from server-side rendering and in the present chapter we shall take a close look at how Angular works, with examples from how the Noesis application was built.

There are two main parts of Angular: its compiler and its runtime. Angular uses TypeScript for development, which is, according to the official website, an “extension of JavaScript” that provides types to the language, turning it into a version that is more similar to Java or C#. The framework also uses many features of ES6, such as classes, that, at the time of this writing, is not fully supported in all browsers. The compiler, thus, is used to transform TypeScript, Angular decorators, and other features into a simpler JavaScript that the runtime can understand.

Angular starts building (“bootstraps”, in their terminology) an application from a root component that it locates, instantiates, and renders. Because of the nature of an SPA, subsequent components will be part of this root. Without getting into too many technical details that are beyond the scope of this paper, each component has a template, an HTML interpolated with special JavaScript-like syntax, that is used to represent what will be rendered in the DOM (Document Object Model). The HTML written in the template is not directly used in rendering. Angular make use of some special functions that parse the template and append the resulting HTML to the DOM. While this may sound slow and expensive in the context of approaches such as React’s Virtual DOM, Angular only does this at the initial creation of the component and then uses an array called Logical View to internally keep track of everything rendered and make changes to the elements faster, as opposed to using the expensive and slower traditional DOM APIs.

Besides the root, Noesis uses 13 components to fill in roles that are expected by an application of its type, such as a list of all books, the details of one particular entry, a component for adding and updating books, for registering and logging in, etc. When accessing the home URL of the application, the user is presented with a simple landing page where they can register, log in or simply enter as guests. Upon hitting “Enter”, they are redirected to the book-list component, which acts as the main hub for the application.
Below the header, the component shows what the top three book categories are (by number of entries) and, for each one, it displays the six best rated books. Then, a list of book cards with some relevant information appears, which gets populated as the user scrolls to the end of the page. To achieve the infinite scrolling effect, the backend uses traditional pagination, fetching a number of entries at a time and remembering how many it has to skip for the next query. In the front end, as with other elements, we use Angular Material43 to append a rotating wheel animation at the end of the list and replace it with the data once it arrives from the server.

Upon clicking on “See More”, the user is redirected to a page with the book’s details, enabled by another component. On the left-hand side of the page, a panel is displayed with some book information such as publisher, number of pages, ISBN, year of publication, and a description. It also shows its current rating with the decimal and yellow stars, its author and the categories it belongs too. The latter are displayed used material chips. Below, a logged user can leave a rating and if they are an admin, the can see the “Admin SuperPowers” box, with two special buttons, one for editing the entry’s details and another for deleting it entirely.

On the right-hand side, we have a picture with the book’s cover, scraped from Amazon, and, below it, logged users can enter and save personalized notes for the book, such as favorite quotes, commentaries and so on. The notes can only be viewed by the logged user and are built with CSS and animations to give them the look of office post-it notes, appear in different colors and get larger and slightly change position on hover. They can also be deleted or edited without leaving this view:

Lastly, also in this view there reside the comment section components, where, naturally, all comment left on this book are displayed for anyone to see. Logged users can participate in the conversation by leaving their own. If they left a comment, two special buttons will appear for it in the table, to delete or to edit it:

It is worth noting that the same component is used for both updating an existing book and adding a new one, since the actions are so similar. As previously mentioned, this feature is restricted to admins and the adding of new books is readily available directly from the navbar. This upsert component uses a pop-up window that it populates with a form, also part of Angular Material, instead of re-directing to a different page. One of the interesting things here is the implementation of how categories and authors are entered. Users can search for existing entries with real time filtering as they type or they can create new entries by typing and hitting the return key.

Data entered in the forms is validated in the backend and, if the operation is unsuccessful, the errors are displayed in red font beneath the afflicted field. Also, front end validation is performed with the client checking user data as they type it in:

Perhaps one of the most appealing aspect of SPAs and frameworks such as Angular is that they are built to detect changes in the application and re-render the UI without the use of refreshing the page. At low level, Angular overrides default event listener DOM APIs to enhance their functionality. All browser events, requests, and other special functions trigger the change detection system which is where the lifecycle hook are called as well, since every component has an associated change detector.

Components are a very powerful tool and the most important feature of the framework. They are complemented by other libraries, as well as additional layers that the developers write, such as a data model and service layer. There is a level of decoupling too, very similar to the one presented in the Noesis backend. Thus, a component does not make HTTP requests directly to server to retrieve the data it needs. Instead it uses a BookService class and defers that responsibility to it. Angular uses RxJS and Observable objects to achieve a-synchronicity in the way it handles requests for data. Observables can be thought of as Futures from other programming languages and, to a lesser extent, Promises in JavaScript. Besides providing an example for this, the snippet bellow also demonstrates (with the HTTP client that is passed to the constructor) that Angular uses Dependency Injection as well. This greatly helps with testing components in isolation.

Single Page Applications are here to stay and, along with React and Vue, Angular is still a strong competitor in the front end frameworks market.

Collaborative Filtering, ALS, and Integrating the Recommendation Engine

Collaborative filtering is a family of machine learning algorithms that are used in recommendation systems. They try to predict whether or not a user will like a specific item based on what previous users liked. The assumption that operates here is that a user is more inclined to like an item that was also favorably reviewed by a second user with whom the first shares interests, rather than one chosen at random.

ALS (Alternating Least Squares) is an implementation of collaborative filtering that can be found in Apache Spark’s ML library. This is an important aspect, since libraries commonly used in ML do not perform well in a distributed environment. ALS can run in parallel across multiple partitions, scales well, and it is considered simpler than alternatives. It also won a prize from Netflix.
We will expose our recommender system as a web API with two endpoints: one for training and one for retrieving recommendations for a specific user ID. Flask has been used for this purpose. Firstly, due to the choice of Python for this microservice since it is the most popular programming language for ML and it is easier to approach than Scala or Java, the other options for Apache Spark. Secondly, because, unlike Django, Flask is much more lightweight and unopinionated. The entire entry point for the application is just 20 lines:

In the train_model() function, data is being read from the MySQL database and then converted to an RDD (Resilient Distributed Dataset), which is a special type of object that Spark uses to help achieve distributed computation. The ALS model is afterwards trained on the RDD and saved locally. When the get_recommendations() function is invoked, it loads the model, passes in the user ID as input and returns a list of recommendations.

The model will be re-trained every time the endpoint /train is requested. The Angular client will reach that endpoint every time a user adds a new rating to any book. This will help ensure that the recommendations provided are as relevant as possible. Of course, the current version of Noesis has random data seeded into it when it comes to user ratings. However, even in this case, after a certain user reviews a couple of books, the recommendations change for the better.

Conclusions

It is clear from the trends at the time of this writing that web applications are going to continue to be amongst the popular choices in software solutions. The advent of technologies such as React Native or Progressive Web Apps seems to support this claim. We have seen in the last chapter how easy it is to integrate a machine learning micro-service in a web application. A development team does not necessarily need the help of a data scientist to optimize hyper parameters in order to reap the benefits of ML. A high level understanding is more than sufficient in order to train a model, as the heavy lifting will fall on the libraries chosen. Since more and more applications are becoming data driven and the possibilities for users to interact are greater than before, data warehousing and data engineering are becoming routine aspects of software architecture for medium and large applications, meaning that the possibility to gain more value through ML-powered insights has risen exponentially in the age of “Big Data”.

Noesis is but a rather simple example of how a portal from book enthusiasts can enrich user experience and retention and can provide valuable insights for business through the integration of a recommendation service. The algorithm used in this paper is a simplified version of how platforms such as YouTube and Netflix managed to grow to such extent by providing personalized suggestions. The age of AI-backed applications is not coming. It is already here.