DocumentDB
Amazon DocumentDB (with MongoDB compatibility) is a fully managed native JSON document database that makes it easy and cost effective to operate critical document workloads at virtually any scale without managing infrastructure.
Elastic Clusters:
Auto-scaling: Elastic Clusters automatically scale storage capacity, compute capacity, and instance count based on your workload demand.
Managed Infrastructure: Amazon DocumentDB manages the underlying infrastructure, including hardware provisioning, software patching, setup, configuration, and backups.
Single Endpoint: You connect to an Elastic Cluster through a single cluster endpoint.
Usage-based Pricing: You pay for the actual resources consumed based on the storage and compute capacity provisioned.
Ideal for Dynamic Workloads: Well-suited for applications with unpredictable workloads that require automatic scaling to handle varying demands.
Instance-based Clusters:
Manual Scaling: You need to manually provision and scale storage capacity and instance count as needed.
Managed Infrastructure: Like Elastic Clusters, DocumentDB manages the underlying infrastructure.
Multiple Endpoints: Instance-based Clusters provide separate reader and writer endpoints.
Fixed Pricing: You pay for provisioned resources regardless of actual usage.
Control Over Resources: Gives you more control over resource allocation but requires manual management.
Clusters
Volume
The cluster's data is stored in the cluster volume with copies in three different Availability Zones.
Amazon DocumentDB 5.0 instance-based clusters support two storage configurations for a database cluster: Amazon DocumentDB standard and Amazon DocumentDB I/O-optimized. For more information see
Notes **
Amazon DocumentDB (with MongoDB compatibility) clusters are deployed within an Amazon Virtual Private Cloud (Amazon VPC). They can be accessed directly by Amazon EC2 instances or other AWS services that are deployed in the same Amazon VPC.
Integration
Outside VPC
Since Amazon DocumentDB is VPC bound we need to create a SSH tunnel and port forwarding to connect to the DB from outside VPC
EC2 security group inbound rules should allow access SSH
Amazon DocumentDB SG inbound rule should allow access to 27107
ssh -i "ec2Access.pem" -L 27017:sample-cluster.node.us-east-1.docdb.amazonaws.com:27017 ubuntu@ec2-34-229-221-164.compute-1.amazonaws.com -N
Above Creates a SSH tunnel
mongosh --sslAllowInvalidHostnames --ssl --sslCAFile global-bundle.pem --username <yourUsername> --password <yourPassword>
Verify by connecting to mongo shell. On successful connection.
Inside VPC
DocumentDB can be accessed directly by Amazon EC2 instances or any other AWS services as long as they are in same VPC
Inside VPC with Lambda
Lambda's Internal VPC:
Lambda functions do run within an internal AWS-managed VPC, but it's not directly visible or configurable by users.
Lambda automatically provisions function inside the VPC and manages this VPC for you.
By default, Lambda functions can access the internet, but they can't access resources inside your VPC.
So we have to move the lambda inside documentDB VPC to make communication between the function and DB.
Once lambda function is moved into the VPC it losses the ability to communicate to the outside world
To make lambda function able to communicate to public internet we need to have a NAT Gateway.
Integrating with Java Spring boot
Connecting Outside VPC
You need to create SSH tunnel if you are connecting outside VPC. I have mentioned command to create tunnel in this document.
In the application.yaml file, we need to make a slight adjustment to the configuration. Specifically, the "host" key should point to localhost, and the "port" key should be configured according to the port forwarding settings.
Connecting within VPC
- In the application.yaml file, we need to make a slight adjustment to the configuration. Specifically, the "host" key should point to documentDB host Since it's in same VPC not need of tunneling
Note:
DocumentDB requires a valid certificate for establishing connections.
Download the certificate from the link below https://truststore.pki.rds.amazonaws.com/global/global-bundle.pem
- application.yaml
spring:
data:
document-db:
user: yourUserName
password: yourPassword
connection-string-template: mongodb://%s:%s@%s:%s/%s?retryWrites=false&directConnection=true&serverSelectionTimeoutMS=2000&tlsAllowInvalidHostnames=true&tls=true
host: yourDocumentDBHost
port: 27017
db-name: yourDatabaseName
DocumentDBConfiguration
package com.platform.assetmanagement.utils.config;
import com.mongodb.ConnectionString;
import com.mongodb.MongoClientSettings;
import com.mongodb.reactivestreams.client.MongoClient;
import com.mongodb.reactivestreams.client.MongoClients;
import jakarta.validation.constraints.NotNull;
import lombok.NonNull;
import lombok.RequiredArgsConstructor;
import lombok.SneakyThrows;
import lombok.extern.slf4j.Slf4j;
import org.apache.http.annotation.Contract;
import org.springframework.beans.factory.annotation.Value;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;
import org.springframework.core.io.ClassPathResource;
import org.springframework.data.mongodb.config.AbstractReactiveMongoConfiguration;
import org.springframework.data.mongodb.core.ReactiveMongoTemplate;
import javax.net.ssl.SSLContext;
import javax.net.ssl.TrustManagerFactory;
import java.io.*;
import java.nio.file.Files;
import java.security.KeyStore;
import java.security.cert.Certificate;
import java.security.cert.CertificateFactory;
import java.util.Set;
import java.util.stream.Collectors;
import java.util.stream.Stream;
@Slf4j
@Configuration
@RequiredArgsConstructor
public class DocumentDBConfig extends AbstractReactiveMongoConfiguration {
private static final String CERT_FILE_PATH = "db-certs/global-bundle.pem";
private static final String END_OF_CERTIFICATE_DELIMITER = "-----END CERTIFICATE-----";
private static final String CERTIFICATE_TYPE = "X.509";
private static final String TLS_PROTOCOL = "TLS";
@Value("${spring.data.document-db.connection-string-template}")
private String connectionStringTemplate;
@Value("${spring.data.document-db.port}")
private String port;
@Value("${spring.data.document-db.db-name}")
private String dbName;
@Value("${spring.data.document-db.host}")
private String host;
@Value("${spring.data.document-db.user}")
private String user;
@Value("${spring.data.document-db.password}")
private String password;
@Override
public MongoClient reactiveMongoClient() {
try {
return MongoClients.create(mongoClientSettings());
} catch (Exception e) {
// Print the error message
System.err.println("Error occurred while creating MongoClient: " + e.getMessage());
// You can also log the error using a logging framework like Logback or Log4j
// logger.error("Error occurred while creating MongoClient", e);
// Rethrow the exception or handle it according to your application's needs
throw new RuntimeException("Error occurred while creating MongoClient", e);
}
}
public MongoClientSettings mongoClientSettings() {
return MongoClientSettings.builder()
.applyConnectionString(new ConnectionString(getConnectionString()))
.applyToSslSettings(builder -> {
builder.enabled(true);
builder.invalidHostNameAllowed(true);
builder.context(createSSLConfiguration());
})
.build();
}
@Bean
public ReactiveMongoTemplate reactiveDocumentTemplate() {
return new ReactiveMongoTemplate(reactiveMongoClient(), getDatabaseName());
}
@SneakyThrows
private SSLContext createSSLConfiguration() {
log.info("Reading AWS PEM certificate...");
// ClassPathResource cpr = new ClassPathResource(CERT_FILE_PATH);
// String certContent = Files.readString(cpr.getFile().toPath());
InputStream inputStream = DocumentDBConfig.class.getClassLoader().getResourceAsStream(CERT_FILE_PATH);
if (inputStream == null) {
throw new FileNotFoundException("PEM file not found: " + CERT_FILE_PATH);
}
String certContent = new BufferedReader(new InputStreamReader(inputStream))
.lines().collect(Collectors.joining("\n"));
Set<String> allCertificates = Stream.of(certContent
.split(END_OF_CERTIFICATE_DELIMITER)).filter(line -> !line.isBlank())
.map(line -> line + END_OF_CERTIFICATE_DELIMITER)
.collect(Collectors.toUnmodifiableSet());
CertificateFactory certificateFactory = CertificateFactory.getInstance(CERTIFICATE_TYPE);
KeyStore keyStore = KeyStore.getInstance(KeyStore.getDefaultType());
keyStore.load(null);
int certNumber = 1;
for (String cert : allCertificates) {
Certificate caCert = certificateFactory.generateCertificate(new ByteArrayInputStream(cert.getBytes()));
keyStore.setCertificateEntry(String.format("AWS-certificate-%s", certNumber++), caCert);
}
TrustManagerFactory trustManagerFactory = TrustManagerFactory.getInstance(TrustManagerFactory.getDefaultAlgorithm());
trustManagerFactory.init(keyStore);
SSLContext sslContext = SSLContext.getInstance(TLS_PROTOCOL);
sslContext.init(null, trustManagerFactory.getTrustManagers(), null);
return sslContext;
}
private String getConnectionString() {
log.info("Generating connection string...");
return String.format(this.connectionStringTemplate,
this.user,
this.password,
this.host,
this.port,
this.getDatabaseName());
}
@Override
@NonNull
protected String getDatabaseName() {
return this.dbName;
}
}
2024-03-25T13:55:24.841+05:30 INFO 30508 --- [ restartedMain] c.p.a.utils.config.DocumentDBConfig : Reading AWS PEM certificate...
2024-03-25T13:55:25.093+05:30 INFO 30508 --- [ restartedMain] org.mongodb.driver.client : MongoClient with metadata {"driver": {"name": "mongo-java-driver|reactive-streams", "version": "4.11.1"}, "os": {"type": "Windows", "name": "Windows 11", "architecture": "amd64", "version": "10.0"}, "platform": "Java/Oracle Corporation/21.0.2+13-LTS-58"} created with settings MongoClientSettings{readPreference=primary, writeConcern=WriteConcern{w=null, wTimeout=null ms, journal=null}, retryWrites=false, retryReads=true, readConcern=ReadConcern{level=null}, credential=MongoCredential{mechanism=null, userName='yourUserName', source='yourDB', password=<hidden>, mechanismProperties=<hidden>}, transportSettings=null, streamFactoryFactory=null, commandListeners=[], codecRegistry=ProvidersCodecRegistry{codecProviders=[ValueCodecProvider{}, BsonValueCodecProvider{}, DBRefCodecProvider{}, DBObjectCodecProvider{}, DocumentCodecProvider{}, CollectionCodecProvider{}, IterableCodecProvider{}, MapCodecProvider{}, GeoJsonCodecProvider{}, GridFSFileCodecProvider{}, Jsr310CodecProvider{}, JsonObjectCodecProvider{}, BsonCodecProvider{}, EnumCodecProvider{}, com.mongodb.client.model.mql.ExpressionCodecProvider@12c05ea9, com.mongodb.Jep395RecordCodecProvider@bb2e83c, com.mongodb.KotlinCodecProvider@23f5c55c]}, loggerSettings=LoggerSettings{maxDocumentLength=1000}, clusterSettings={hosts=[localhost:27017], srvServiceName=mongodb, mode=SINGLE, requiredClusterType=UNKNOWN, requiredReplicaSetName='null', serverSelector='null', clusterListeners='[]', serverSelectionTimeout='2000 ms', localThreshold='15 ms'}, socketSettings=SocketSettings{connectTimeoutMS=10000, readTimeoutMS=0, receiveBufferSize=0, proxySettings=ProxySettings{host=null, port=null, username=null, password=null}}, heartbeatSocketSettings=SocketSettings{connectTimeoutMS=10000, readTimeoutMS=10000, receiveBufferSize=0, proxySettings=ProxySettings{host=null, port=null, username=null, password=null}}, connectionPoolSettings=ConnectionPoolSettings{maxSize=100, minSize=0, maxWaitTimeMS=120000, maxConnectionLifeTimeMS=0, maxConnectionIdleTimeMS=0, maintenanceInitialDelayMS=0, maintenanceFrequencyMS=60000, connectionPoolListeners=[], maxConnecting=2}, serverSettings=ServerSettings{heartbeatFrequencyMS=10000, minHeartbeatFrequencyMS=500, serverListeners='[]', serverMonitorListeners='[]'}, sslSettings=SslSettings{enabled=true, invalidHostNameAllowed=true, context=javax.net.ssl.SSLContext@19d683bd}, applicationName='null', compressorList=[], uuidRepresentation=UNSPECIFIED, serverApi=null, autoEncryptionSettings=null, dnsClient=null, inetAddressResolver=null, contextProvider=null}
2024-03-25T13:55:25.523+05:30 INFO 30508 --- [ restartedMain] o.s.b.d.a.OptionalLiveReloadServer : LiveReload server is running on port 35729
2024-03-25T13:55:25.898+05:30 INFO 30508 --- [localhost:27017] org.mongodb.driver.cluster : Monitor thread successfully connected to server with description ServerDescription{address=localhost:27017, type=REPLICA_SET_PRIMARY, state=CONNECTED, ok=true, minWireVersion=0, maxWireVersion=13, maxDocumentSize=16777216, logicalSessionTimeoutMinutes=30, roundTripTimeNanos=541759900, setName='rs0', canonicalAddress=platformasset.ccouih7cdiv2.ap-northeast-1.docdb.amazonaws.com:27017, hosts=[platformasset.ccouih7cdiv2.ap-northeast-1.docdb.amazonaws.com:27017], passives=[], arbiters=[], primary='platformasset.ccouih7cdiv2.ap-northeast-1.docdb.amazonaws.com:27017', tagSet=TagSet{[]}, electionId=7fffffff0000000000000001, setVersion=null, topologyVersion=null, lastWriteDate=Mon Mar 25 13:54:54 IST 2024, lastUpdateTimeNanos=442201381626300}
2024-03-25T13:55:26.065+05:30 INFO 30508 --- [ restartedMain] c.p.a.utils.config.DocumentDBConfig : Generating connection string...
2024-03-25T13:55:26.065+05:30 INFO 30508 --- [ restartedMain] c.p.a.utils.config.DocumentDBConfig : Reading AWS PEM certificate...
2024-03-25T13:55:26.090+05:30 INFO 30508 --- [ restartedMain] org.mongodb.driver.client : MongoClient with metadata {"driver": {"name": "mongo-java-driver|reactive-streams", "version": "4.11.1"}, "os": {"type": "Windows", "name": "Windows 11", "architecture": "amd64", "version": "10.0"}, "platform": "Java/Oracle Corporation/21.0.2+13-LTS-58"} created with settings MongoClientSettings{readPreference=primary, writeConcern=WriteConcern{w=null, wTimeout=null ms, journal=null}, retryWrites=false, retryReads=true, readConcern=ReadConcern{level=null}, credential=MongoCredential{mechanism=null, userName='yourUserName', source='yourDB', password=<hidden>, mechanismProperties=<hidden>}, transportSettings=null, streamFactoryFactory=null, commandListeners=[], codecRegistry=ProvidersCodecRegistry{codecProviders=[ValueCodecProvider{}, BsonValueCodecProvider{}, DBRefCodecProvider{}, DBObjectCodecProvider{}, DocumentCodecProvider{}, CollectionCodecProvider{}, IterableCodecProvider{}, MapCodecProvider{}, GeoJsonCodecProvider{}, GridFSFileCodecProvider{}, Jsr310CodecProvider{}, JsonObjectCodecProvider{}, BsonCodecProvider{}, EnumCodecProvider{}, com.mongodb.client.model.mql.ExpressionCodecProvider@12c05ea9, com.mongodb.Jep395RecordCodecProvider@bb2e83c, com.mongodb.KotlinCodecProvider@23f5c55c]}, loggerSettings=LoggerSettings{maxDocumentLength=1000}, clusterSettings={hosts=[localhost:27017], srvServiceName=mongodb, mode=SINGLE, requiredClusterType=UNKNOWN, requiredReplicaSetName='null', serverSelector='null', clusterListeners='[]', serverSelectionTimeout='2000 ms', localThreshold='15 ms'}, socketSettings=SocketSettings{connectTimeoutMS=10000, readTimeoutMS=0, receiveBufferSize=0, proxySettings=ProxySettings{host=null, port=null, username=null, password=null}}, heartbeatSocketSettings=SocketSettings{connectTimeoutMS=10000, readTimeoutMS=10000, receiveBufferSize=0, proxySettings=ProxySettings{host=null, port=null, username=null, password=null}}, connectionPoolSettings=ConnectionPoolSettings{maxSize=100, minSize=0, maxWaitTimeMS=120000, maxConnectionLifeTimeMS=0, maxConnectionIdleTimeMS=0, maintenanceInitialDelayMS=0, maintenanceFrequencyMS=60000, connectionPoolListeners=[], maxConnecting=2}, serverSettings=ServerSettings{heartbeatFrequencyMS=10000, minHeartbeatFrequencyMS=500, serverListeners='[]', serverMonitorListeners='[]'}, sslSettings=SslSettings{enabled=true, invalidHostNameAllowed=true, context=javax.net.ssl.SSLContext@7fdd661e}, applicationName='null', compressorList=[], uuidRepresentation=UNSPECIFIED, serverApi=null, autoEncryptionSettings=null, dnsClient=null, inetAddressResolver=null, contextProvider=null}
2024-03-25T13:55:26.634+05:30 INFO 30508 --- [localhost:27017] org.mongodb.driver.cluster : Monitor thread successfully connected to server with description ServerDescription{address=localhost:27017, type=REPLICA_SET_PRIMARY, state=CONNECTED, ok=true, minWireVersion=0, maxWireVersion=13, maxDocumentSize=16777216, logicalSessionTimeoutMinutes=30, roundTripTimeNanos=406052800, setName='rs0', canonicalAddress=DocumentDB.ccouih7cdiv2.ap-northeast-1.docdb.amazonaws.com:27017, hosts=[Document.ccouih7cdiv2.ap-northeast-1.docdb.amazonaws.com:27017], passives=[], arbiters=[], primary='Document.ccouih7cdiv2.ap-northeast-1.docdb.amazonaws.com:27017', tagSet=TagSet{[]}, electionId=7fffffff0000000000000001, setVersion=null, topologyVersion=null, lastWriteDate=Mon Mar 25 13:54:55 IST 2024, lastUpdateTimeNanos=442202124708600}
Now that the connection has established we can do simple crud operation.
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-data-mongodb</artifactId>
</dependency>
UserEntity
import org.springframework.data.annotation.Id;
import org.springframework.data.mongodb.core.mapping.Document;
@Document(collection = "users")
public class User {
@Id
private String id;
private String name;
private int age;
// Getters and setters
}
UserRepository
import org.springframework.data.mongodb.repository.MongoRepository;
public interface UserRepository extends MongoRepository<User, String> {
}
UserService
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.stereotype.Service;
import java.util.List;
@Service
public class UserService {
@Autowired
private UserRepository userRepository;
public List<User> getAllUsers() {
return userRepository.findAll();
}
public User getUserById(String id) {
return userRepository.findById(id).orElse(null);
}
public User createUser(User user) {
return userRepository.save(user);
}
public User updateUser(String id, User newUser) {
newUser.setId(id);
return userRepository.save(newUser);
}
public void deleteUser(String id) {
userRepository.deleteById(id);
}
}
UserController
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.web.bind.annotation.*;
import java.util.List;
@RestController
@RequestMapping("/api/users")
public class UserController {
@Autowired
private UserService userService;
@GetMapping
public List<User> getAllUsers() {
return userService.getAllUsers();
}
@GetMapping("/{id}")
public User getUserById(@PathVariable String id) {
return userService.getUserById(id);
}
@PostMapping
public User createUser(@RequestBody User user) {
return userService.createUser(user);
}
@PutMapping("/{id}")
public User updateUser(@PathVariable String id, @RequestBody User user) {
return userService.updateUser(id, user);
}
@DeleteMapping("/{id}")
public void deleteUser(@PathVariable String id) {
userService.deleteUser(id);
}
}
As you the see API is successfully creating document in the Document DB cluster.
Conclusion
In summary, integrating Amazon DocumentDB with Java Spring Boot and Amazon Lambda provides developers with a powerful and streamlined solution for building cloud-native applications. This combination offers compatibility, performance, simplified development, scalability, and cost-effectiveness, enabling developers to create resilient, scalable, and cost-efficient applications that meet modern business requirements.