First, I’ll cover some anatomy and physiology of the heart.
Cardiac anatomy

The heart is divided into two sides. The right side (on the left of the image) receives blood from the body and pumps it to the lungs. The left side (on the right of the image) receives blood back from the lungs and pumps it to the body.
Cardiac physiology
Blood entering the heart first enters an atrium. This contracts to fill the ventricle (the main chamber), which then contracts to expel blood from the heart. This cycle is divided into two main parts: diastole (when the atrium contracts and the ventricle is relaxed and filling) and systole (when the ventricle contracts).
Left ventricular systolic pressure must exceed the aortic diastolic pressure for blood to be ejected from the ventricle.
Blood pressure
The cardiac output (CO - measured in litres per minute) depends on the stroke volume (SV - blood pumped with each cardiac cycle) multiplied by the heart rate (HR).
Systemic vascular resistance (SVR - often also called total peripheral resistance) is equal to the difference between the mean arterial pressure (MAP) and central venous pressure (CVP) divided by the cardiac output. CVP is usually close to zero, so this is often shortened to:

Blood pressure (BP) is a measure of the force being exerted on the walls of arteries as blood is pumped out of the heart. It is proportional to cardiac output (CO) and systemic vascular resistance (SVR). It is usually expressed as the systolic blood pressure over the diastolic blood pressure.
Importantly, with regard to your question, while peripheral systolic blood pressure is largely due to ventricular contraction in the heart, diastolic pressure in peripheral arteries is due to the walls of the arteries contracting passively after having been expanded during the pressure wave of systole.
This diagram shows the atrial, ventricular and aortic pressures during the cardiac cycle.

This is a pressure volume loop showing the relationship of ventricular pressure and volume during a single cardiac cycle.

You can see from both of these images that there is a point in early diastole where left ventricular pressure drops close to zero, before rising slowly. Later in diastole, the pressure rises gradually and the left ventricular end-diastolic pressure (LVEDP) is commonly measured as a metric of cardiac performance.
Why the low left ventricular diastolic pressure?
This low diastolic ventricular pressure is primarily because a well-functioning ventricle will now be isolated from the systemic circulation by the closed aortic valve. In addition, it will have expelled a significant amount of the blood contained within (a normal ejection fraction (EF) is 55-70%), it is now relaxing and expanding in size and it has not yet received much blood from the atrium.
In heart failure with a impaired systolic function, the ejection fraction (EF) drops, but there is a phenomenon of diastolic dysfunction, where the EF is normal and the problem is impaired relaxation and compliance of the left ventricle. This is commonly linked to hypertension, as displayed in this paper.
This pressure volume loop (in red) demonstrates this effect in diastolic dysfunction. ESPVR / EDPVR = end systolic / diastolic pressure volume relationship (a gradient of the graph or first derivative of pressure with respect to volume).

Also, there is evidence of a link between a higher LVEDP and increased mortality when measured during coronary angiography for acute myocardial infarction.
In summary, the left ventricular diastolic pressure being low is normal and several factors are responsible for that. It is higher where there is ventricular diastolic dysfunction.
Sources:
Source for images
CV Physiology
Planer et al
Lelande and Johnson
Bagai et al