504 Gateway Time-out and targetPendingRequests

[mikaelasanchez]

Hello! I hope I'm posting this in the right place.

Gateway Timeout

I'm trying to get a simple Hello World app working with KEDA and followed the instructions in the walkthrough page
Firstly, when I edit the ingress to point to keda-add-ons-http-interceptor-proxy, the pod starts up (good)
But then I receive a 504 Gateway Time-out error, instead of it showing "Hello World"
After a while, it does scale back to 0 (good)

Not sure what's going on here?

targetPendingRequests

In addition, I'm getting this error:

My YAML file:

kind: HTTPScaledObject
apiVersion: http.keda.sh/v1alpha1
metadata:
    name: hello
spec:
    host: hello.127.0.0.1.nip.io
    scaleTargetRef:
        deployment: hello
        service: hello
        port: 80
        targetPendingRequests: 10

Thanks in advance :)

[arschles]

Hi @mikaelasanchez thanks for reporting this!

For the targetPendingRequests problem, are you installing the v0.2.0 chart? If so, can you run kubectl get crd httpscaledobjects.http.keda.sh -o yaml and paste the result? That will show the schema so we can see if/why targetPendingRequests is not there.

For the 504 problem, are you routing traffic through a cloud load balancer of some kind? The first request can take some time to spin up the pod (as you saw) and while the interceptor waits, other systems that sit in the network path upstream from it might not.

[avinash31d]

Hi, @arschles
I am facing the same issue, and ran kubectl get crd httpscaledobjects.http.keda.sh -o yaml

here is the result, I dont see targetPendingRequests

`
apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata:
annotations:
controller-gen.kubebuilder.io/version: v0.6.1
meta.helm.sh/release-name: http-add-on
meta.helm.sh/release-namespace: keda
creationTimestamp: "2022-01-19T11:02:52Z"
generation: 1
labels:
app.kubernetes.io/managed-by: Helm
name: httpscaledobjects.http.keda.sh
resourceVersion: "113085"
uid: 5185bccb-e256-4c02-b5a9-fdac3ec29337
spec:
conversion:
strategy: None
group: http.keda.sh
names:
kind: HTTPScaledObject
listKind: HTTPScaledObjectList
plural: httpscaledobjects
shortNames:
- httpso
singular: httpscaledobject
scope: Namespaced
versions:

• additionalPrinterColumns:
  • jsonPath: .spec.scaleTargetRef.deploymentName
    name: ScaleTargetDeploymentName
    type: string
  • jsonPath: .spec.scaleTargetRef
    name: ScaleTargetServiceName
    type: string
  • jsonPath: .spec.scaleTargetRef
    name: ScaleTargetPort
    type: integer
  • jsonPath: .spec.replicas.min
    name: MinReplicas
    type: integer
  • jsonPath: .spec.replicas.max
    name: MaxReplicas
    type: integer
  • jsonPath: .metadata.creationTimestamp
    name: Age
    type: date
  • jsonPath: .status.conditions[?(@.type=="HTTPScaledObjectIsReady")].status
    name: Active
    type: string
    name: v1alpha1
    schema:
    openAPIV3Schema:
    properties:
    apiVersion:
    description: 'APIVersion defines the versioned schema of this representation
    of an object. Servers should convert recognized schemas to the latest
    internal value, and may reject unrecognized values. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources'
    type: string
    kind:
    description: 'Kind is a string value representing the REST resource this
    object represents. Servers may infer this from the endpoint the client
    submits requests to. Cannot be updated. In CamelCase. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds'
    type: string
    metadata:
    type: object
    spec:
    description: HTTPScaledObjectSpec defines the desired state of HTTPScaledObject
    properties:
    host:
    description: The host to route. All requests with this host in the
    "Host" header will be routed to the Service and Port specified in
    the scaleTargetRef
    type: string
    replicas:
    description: (optional) Replica information
    properties:
    max:
    description: Maximum amount of replicas to have in the deployment
    (Default 100)
    format: int32
    type: integer
    min:
    description: Minimum amount of replicas to have in the deployment
    (Default 0)
    format: int32
    type: integer
    type: object
    scaleTargetRef:
    description: The name of the deployment to route HTTP requests to
    (and to autoscale). Either this or Image must be set
    properties:
    deployment:
    description: The name of the deployment to scale according to
    HTTP traffic
    type: string
    port:
    description: The port to route to
    format: int32
    type: integer
    service:
    description: The name of the service to route to
    type: string
    required:
    - deployment
    - port
    - service
    type: object
    required:
    - host
    - scaleTargetRef
    type: object
    status:
    description: HTTPScaledObjectStatus defines the observed state of HTTPScaledObject
    properties:
    conditions:
    description: List of auditable conditions of the operator
    items:
    description: Condition to store the condition state
    properties:
    message:
    description: A human readable message indicating details about
    the transition.
    type: string
    reason:
    description: The reason for the condition's last transition.
    enum:
    - ErrorCreatingAppScaledObject
    - AppScaledObjectCreated
    - TerminatingResources
    - AppScaledObjectTerminated
    - AppScaledObjectTerminationError
    - PendingCreation
    - HTTPScaledObjectIsReady
    type: string
    status:
    description: Status of the condition, one of True, False, Unknown.
    type: string
    timestamp:
    description: Timestamp of the condition
    type: string
    type:
    description: Type of condition
    enum:
    - Created
    - Error
    - Pending
    - Unknown
    - Terminating
    - Terminated
    - Ready
    type: string
    required:
    - status
    - type
    type: object
    type: array
    type: object
    type: object
    served: true
    storage: true
    subresources:
    status: {}
    status:
    acceptedNames:
    kind: HTTPScaledObject
    listKind: HTTPScaledObjectList
    plural: httpscaledobjects
    shortNames:
  • httpso
    singular: httpscaledobject
    conditions:
• lastTransitionTime: "2022-01-19T11:02:52Z"
  message: no conflicts found
  reason: NoConflicts
  status: "True"
  type: NamesAccepted
• lastTransitionTime: "2022-01-19T11:02:52Z"
  message: the initial names have been accepted
  reason: InitialNamesAccepted
  status: "True"
  type: Established
  storedVersions:
• v1alpha1
  `

[arschles]

@avinash31d what version of the http add on do you have installed?

[avinash31d]

@arschles

It is the latest stable release version which is 0.2.0,
i used the below command to install it by referring this install doc

helm install http-add-on kedacore/keda-add-ons-http --namespace ${NAMESPACE}

[arschles]

@avinash31d can you try this:

1. Delete the HTTP Addon helm release from your cluster: helm delete -n ${NAMESPACE} <HTTP addon name>
2. Delete the HTTPScaledObject CRD: kubectl delete crd httpscaledobject
3. Then re-install the HTTP Addon according to the same install document you referenced (https://github.com/kedacore/http-add-on/blob/main/docs/install.md)

Also, can you change your HTTPScaledObject yaml file to this:

kind: HTTPScaledObject
apiVersion: http.keda.sh/v1alpha1
metadata:
    name: hello
spec:
    host: hello.127.0.0.1.nip.io
    targetPendingRequests: 10 # here
    scaleTargetRef:
        deployment: hello
        service: hello
        port: 80
        # targetPendingRequests: 10 # not here

(there was a bug in the reference documentation for this targetPendingRequests field. fix is here: #372)

[avinash31d]

@arschles
I tried the above mentioned steps but no luck

after deleting and reinstalling the http-add-on here is the output, I think the version is correct

NAME            NAMESPACE       REVISION        UPDATED                                                STATUS              CHART                         APP VERSION
http-add-on     keda                     1               2022-01-21 10:20:59.866086309 +0530 IST   deployed        keda-add-ons-http-0.2.1    0.2.0      

Also below is my HTTPScaledObject yaml file

kind: HTTPScaledObject
apiVersion: http.keda.sh/v1alpha1
metadata:
    name: {{ include "httptestapi.fullname" . }}
spec:
    host: {{ .Values.host }}
    targetPendingRequests: {{ .Values.targetPendingRequests }}
    scaleTargetRef:
        deployment: {{ include "httptestapi.fullname" . }}
        service: {{ include "httptestapi.fullname" . }}
        port: 8080
    replicas:
        min: {{ .Values.autoscaling.http.minReplicas }}
        max: {{ .Values.autoscaling.http.maxReplicas }}

Error: INSTALLATION FAILED: unable to build kubernetes objects from release manifest: error validating "": error validating data: ValidationError(HTTPScaledObject.spec): unknown field "targetPendingRequests" in sh.keda.http.v1alpha1.HTTPScaledObject.spec

[avinash31d]

@arschles
Also I checked the crd here https://github.com/kedacore/charts/blob/d1ea02c79b5c8d2b78ebfc95830f1fbbe5b7d7d8/http-add-on/templates/crd.yaml

but i dont see targetPendingRequests

[avinash31d]

@arschles
Seems like targetPendingRequests property is missing int crd.yaml of the latest chart versions 0.2.0 or 0.2.1

I updated it and created a new PR kedacore/charts#230, please review it and merge if useful.

(This is my first PR in any opensource project, please let me know if I missed something, I will fix it)

I followed the guidelines here [https://github.com/kedacore/charts/blob/main/CONTRIBUTING.md](contributing to charts)

[togrulazizli]

@arschles I am having same 504 error problem. Scaling up and down part works perfectly but once it scales up (0 to 1) and pod is ready (containers are all running), I am receiving 504 error and it happens no matter how many times I refresh the http request. Once I change the ingress from interceptor service to deployment service then it starts to work.
It seems for some reason when web server receives requests from interceptor, it does not reply.
server is nginx web server, now I don't know maybe interceptor changes something in requests while redirecting them to web server and web server does not reply.

I am using 0.3.0 version of http-add-on app
Any help is appreciated :)