This is a very broad question. Addressing all aspects may result in an answer that is too long. Therefore only some examples can be looked at.
Since the premise starts with chicken pox vaccine as an example for a successful vaccine this answer starts with an even more successful vaccine:
Smallpox was the most dreaded of infections, especially in the 18th century when it seemed to be particularly virulent. The smallpox virus is very large and complex as viruses go and there are numerous related viruses which infect animals and are more or less analogous to the smallpox, the symptoms of which are characterised by the eruption of pimples, blisters and pustules. The smallpox virus itself can be classified into two distinct types, one of normal virulence (Variola major) and the other an attenuated form (Variola minor). This last point is important because it explains, at least partly, the apparent confusion over the years between the smallpox virus and the vaccine (vaccinia or cowpox virus), at a time when the distinction was made solely on the basis of clinical symptoms observed in patients or on attempts to transmit it between species; for example, between man and cow for the smallpox virus. At that time, the "germ" of smallpox was considered to be innate among humans, or almost so, and manifested itself in a so-called crisis by its "evacuation" from the human organism.
From Hervé Bazin: "The Eradication of Smallpox: Edward Jenner and the First and Only Eradication of a Human Infectious Disease", Academic Press: Cambridge, 1999, p4.
This passage about the history of the smallpox vaccine illustrates some points necessary for a vaccine to be developed. We need to understand the viral agent, the human immune response, and we need to find a "form of the viral agent" that elicits a reliable but comparatively mild immune response that leads to immunity later.
There are different methods to arrive at such a vaccine:
Live attenuated (smallpox) – killed whole organisms (cholera) – Purified proteins of organisms or polysaccharides (tetanus) – reassortants (influenza, live and killed) – Genetically engineered (hepatitis B recombinant).
Cf. Stanley A. Plotkin: "History of Vaccine Development", Springer: New York, Dordrecht, 2011, p3.
Not all of these methods are suitable for all viruses. Some viruses are difficult to attenuate, some difficult to grow in cultivation, some are changing so fast that a successful immune response does lead to immunity but only to the agent used and not for the agents encountered later on, almost nullifying the results obtained.
Smallpox is not only large but also quite stable over time and between hosts so that a human immune system can acquire immunity, even if the very similar and easy to obtain cowpox is used as the source to get the required antigens – all four variants of orthopoxviruses confer cross-immunity.
Looking at the "unsuccessful" side of the table, the rhinoviruses
are currently around 160 recognized types of human rhinoviruses that differ according to their surface proteins (serotypes). They are lytic in nature and are among the smallest viruses, with diameters of about 30 nanometers. […]
There are no vaccines against these viruses as there is little-to-no cross-protection between serotypes. At least 99 serotypes of Human rhinoviruses affecting humans have been sequenced…
Meaning that if an immunity develops to one type, at least theoretically 160 more infections are left for a single human on the menu.
With HIV the story is similar but different in that the virus can almost be called 'unstable':
The environment of HIV in vivo is not static, but dynamic and reactive, so topographical imagery is dubious. Also unlike clas- sical genetics, we must distinguish two contributions to viral fitness: one relating to the ability of HIV to grow in its target cells, independent of any immune response, and another reflecting immune pressure.[…]
Perhaps the most striking fact about HIV in vivo is its extraordinary repli- cation rate. Unlike some viruses (such as chickenpox or herpes), HIV never enters a dormant or “latent” stage, but reproduces continuously in the body over the whole time-course of infection. The number of PITs in the chronic phase is in the range 10–100 million, and the turnover time is 2–4 days. HIV’s mutation rate is also remarkable: it is at least five orders-of-magnitude higher than for DNA-bearing, eukaryotic organisms. The rate was measured in the early 1990s in the test tube, with HIV propagating in immortalized T-cell lines (again, biologists prefer the Latin, and refer to the observation as “in vitro”, literally in glass) and yielded the average figure: about 0.3 changes per genome per replication cycle.
W. David Wick & Otto O. Yang: "War in the Body: The Evolutionary Race Between HIV and the Human Immune System and the Implications for Vaccines", Springer: New York: Heidelberg, 2013.
That is a very fast moving target! But he outlook is not as grim as these above numbers might suggest:
We believe that every major disease will eventually have its vaccine. However, if we consider major infectious agents, such as human immunodeficiency virus (HIV), hepatitis C virus (HCV), and Malaria, despite many years of effort, billions of dollars spent and countless animal lives sacrificed, no vaccine is available to protect against these infections. What prevents us from being victorious?
- The genetic diversity of the target pathogen. (In RNA viruses such as HIV and HCV, the error prone RNA dependent polymerase generates quasispecies. In addition, influenza vaccines need to be reformulated annually, due to antigenic drift)
- The discrepancy between immunogenicity and protection (many HIV vaccine candidates induce strong T and B cell responses in pre-clinical and Phase-I trials, these responses have thus far failed to correlate with protection in larger scale trials.)
- Vector or Immunogen, which one matters? (While an effective vaccine may need to be multivalent, comprising multiple alleles for a given polymorphic antigen, and/or the antigen derived from conserved regions, the delivery vectors are at least as important as the immunogen itself. The vectors modulate innate and adaptive immunity, hopefully enabling the vaccine antigen to elicit the right response.)
- The discrepancy between local and systemic responses. (…of the vectors used in HIV vaccine design, viral vectors with mucosal tropism, e.g., adenoviruses and influenza viruses, are particularly interesting, given that genitorectal mucosa is the first site of contact in HIV transmission. Most systemic vaccines do not elicit mucosal responses, and it is uncertain if mucosal delivery of antigen can induce systemic immunity.)
- Infant vaccination, how much do we know? (The innate immune system does not reach full capacity until the teenage years, and as adaptive immunity in newborns is intrinsically skewed to a Th2-type, the neonatal and infant immune responses to many vaccines are suboptimal.)
- Immune subversion and immunosuppression. (Malaria-infected red blood cells have an amazing capacity to induce FOXP3+ expression, a marker of highly suppressive regulatory T cells (Treg), on co-cultured autologous T cells, suggesting that widespread induction in vivo would not require direct contact with the parasite.)
Shuo Li, Magdalena Plebanski et al.: "Why vaccines to HIV, HCV and Malaria have so far failed - challenges to developing vaccines against immunoregulating pathogens", Frontiers in Microbiolog and Frontiers in Immunology, 2016, DOI:10.3389/fmicb.2015.01318